SentSim: Crosslingual Semantic Evaluation of Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Machine translation (MT) is currently evaluated in one of two ways: monolingually or trained crosslingually by building a supervised model to predict quality scores from human-labeled data. |
| Approach: | They propose an unsupervised model that directly compares the source and machine translated sentence using strong pretrained multilingual word and sentence representations. |
| Outcome: | The proposed model outperforms glass-box approaches to quality estimation that rely on a supervised model. |
Similar Papers
Revisiting Machine Translation for Cross-lingual Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in cross-lingual learning has pivoted around multilingual models, which are typically pretrained on unlabeled corpora in multiple languages using some form of language modeling objective. |
| Approach: | They propose to use a stronger machine translation system to mitigat mismatch between training on original text and running inference on machine translated text. |
| Outcome: | The proposed approach is highly task dependent and calls into question the dominance of multilingual models for cross-lingual classification. |
On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation (2020.acl-main)
Copied to clipboard
| Challenge: | a standard evaluation setup for supervised machine learning tasks does not hold for natural language generation tasks. |
| Approach: | They propose to use reference-free machine translation evaluation to compare source texts to system translations to find key limitations. |
| Outcome: | The proposed metrics perform poorly as semantic encoders for reference-free machine translation evaluation. |
Can Machine Translation Bridge Multilingual Pretraining and Cross-lingual Transfer Learning? (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models that pretrain for cross-lingual tasks do not improve cross-linguistic learning. |
| Approach: | They propose to employ machine translation as a continued training objective to enhance language representation learning by bridging multilingual pretraining and cross-lingual applications. |
| Outcome: | The proposed model performance is compared with existing models and their latent representations. |
On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation (2022.naacl-main)
Copied to clipboard
| Challenge: | Modern unsupervised machine translation systems reach reasonable translation quality under clean and controlled data conditions. |
| Approach: | They compare unsupervised and supervised machine translation systems of similar quality . they combine the benefits of both methods into a single system . |
| Outcome: | The proposed system improves adequacy and fluency as measured by human evaluators. |
Beyond Reference: Evaluating High Quality Translations Better than Human References (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing machine translation metrics give maximum score to reference sentence . however, these metrics overlook the possibility that candidate sentences outperform reference sentences in terms of quality. |
| Approach: | They propose a machine translation metrics that give an absolute score to a translated sentence based on the similarity with the reference sentence. |
| Outcome: | The proposed measure outperforms existing MT metrics in terms of quality and assigns positive scores to candidates that outperformed reference sentences. |
Human or Neural Translation? (2020.coling-main)
Copied to clipboard
Shivendra Bhardwaj, David Alfonso Hermelo, Phillippe Langlais, Gabriel Bernier-Colborne, Cyril Goutte, Michel Simard
| Challenge: | a recent study shows that deep neural models have improved machine translation . identifying machine translation is still feasible, but is not yet known. |
| Approach: | They train and apply deep neural models to distinguish between human and machine translations . they use a monolingual and bilingual task to train and train 18 classifiers based on their results . |
| Outcome: | The proposed model improves the ability to distinguish between human and machine translations at the sentence level. |
DEMETR: Diagnosing Evaluation Metrics for Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | BLEU scores are based on string overlap, but they are opaque in comparison to newer learned metrics. |
| Approach: | They propose a dataset to evaluate MT evaluation metrics based on linguistic perturbations in English . they find learned metrics perform substantially better than string-based metrics . |
| Outcome: | The proposed dataset shows that learned metrics perform better than string-based metrics . the dataset contains 31K English examples that cover 35 different linguistic phenomena . |
Multi-Dimensional Machine Translation Evaluation: Model Evaluation and Resource for Korean (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on MT evaluation characterize quality of output with a single number . a recent advancement in MT technologies has enabled higher-quality, more nuanced translations . |
| Approach: | They propose a 1200-sentence MQM evaluation benchmark for English-Korean and a reference-free QE setup to evaluate the quality of the translations. |
| Outcome: | The proposed model outperforms the existing model in style and accuracy. |
Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation (P19-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics are limited and can be easily portable to new languages. |
| Approach: | They propose a simple unsupervised metric and additional supervised metrics which rely on contextual word embeddings to encode the translation and reference sentences. |
| Outcome: | The proposed model outperforms existing metrics on the WMT 2017 dataset and is more accurate than existing models. |
Word Embedding-Based Automatic MT Evaluation Metric using Word Position Information (N19-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics for machine translation are difficult to address word meaning because it is a surface-level metric. |
| Approach: | They propose to use word embeddings, sentence-level tf-idf, and cosine similarity between two word embeds as features, weight, and the distance between two features as features. |
| Outcome: | The proposed metric can evaluate machine translation based on word meaning . it achieves highest correlation with human judgment among several representative metrics. |