Papers by James Cross
Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications (2021.emnlp-main)
Copied to clipboard
| Challenge: | Sentence-level Quality estimation (QE) is traditionally a regression task . but large multilingual contextualized language models are expensive and infeasible for real-world applications. |
| Approach: | They evaluate several model compression techniques for QE and find they are inefficient . they argue that a full model parameterization is required to achieve SoTA results . |
| Outcome: | The proposed models are poorly expressive in a regression task, the authors argue . they show that reframing QE as a classification problem and evaluating models would improve their performance in real-world applications. |
Alternative Input Signals Ease Transfer in Multilingual Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work in multilingual machine translation (MMT) has focused on the potential of positive transfer between languages. |
| Approach: | They propose to augment training data with alternative signals that unify different writing systems, such as phonetic, romanized, and transliterated input. |
| Outcome: | The proposed model outperforms strong ensemble baselines on Indic and Turkic languages by 1.3 BLEU points on both languages. |
Improving Zero-Shot Translation by Disentangling Positional Information (2021.acl-long)
Copied to clipboard
| Challenge: | Multilingual neural machine translation has shown the capability of directly translating between language pairs unseen in training, i.e. zero-shot translation. |
| Approach: | They propose to remove residual connections in an encoder layer to reduce the difficulty of generalizing to new translation directions. |
| Outcome: | The proposed model outperforms pivot-based translation in terms of quality and ease of integration of new languages. |
Multilingual Machine Translation with Hyper-Adapters (2022.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual machine translation suffers from negative interference across languages. |
| Approach: | They propose a rescaling fix that reduces the number of parameters and enables training larger hyper-networks. |
| Outcome: | The proposed approach outperforms regular adapters and achieves the same performance with 12 times less parameters. |
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work on multilingual pre-trained models has focused on pre-training transformers on concatenated corpora of a large number of languages. |
| Approach: | They propose a language-specific module approach that allows for more languages to be trained post-hoc. |
| Outcome: | The proposed model can be pre-trained on multiple languages with no drop in performance . |
Data Selection Curriculum for Neural Machine Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Neural Machine Translation models are typically trained on heterogeneous data that are concatenated and randomly shuffled. |
| Approach: | They propose a two-stage curriculum training framework where a NMT model is fine-tuned on subsets of data, selected by deterministic scoring and online scoring. |
| Outcome: | The proposed framework improves on six language pairs comprising low- and high-resource languages and shows up to +2.2 BLEU improvement and faster convergence. |
Tricks for Training Sparse Translation Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Multitask learning with an unbalanced data distribution skews model learning towards high resource tasks. |
| Approach: | They propose to use a temperature heating mechanism and dense pre-training to mitigate this by training models with a fixed model capacity. |
| Outcome: | The proposed techniques improve performance on two multilingual translation benchmarks compared to BASELayers and Dense scaling baselines and in combination, more than 2x model convergence speed. |
Multilingual Neural Machine Translation with Deep Encoder and Multiple Shallow Decoders (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent work in multilingual translation has improved translation quality surpassing bilingual baselines using deep transformer models with increased capacity. |
| Approach: | They propose a deep encoder with multiple shallow decoders to reduce inference latency while maintaining translation quality. |
| Outcome: | The proposed model achieves 1.8x speedup on average compared to a standard transformer model with no drop in translation quality. |
XLEnt: Mining a Large Cross-lingual Entity Dataset with Lexical-Semantic-Phonetic Word Alignment (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generate named entity lexica for lower-resource languages are under performing. |
| Approach: | They propose a technique to automatically mine cross-lingual named-entity lexica from mined web data. |
| Outcome: | The proposed technique outperforms baselines at extracting cross-lingual entity pairs and mines 164 million entity pairs from 120 different languages aligned with English. |
Efficiently Upgrading Multilingual Machine Translation Models to Support More Languages (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing multilingual machine translation models need to be upgraded as data becomes available in more languages. |
| Approach: | They propose three techniques that speed up the effective learning of new languages and alleviate catastrophic forgetting . |
| Outcome: | The proposed techniques exceed the performance of a same-sized baseline model with 30% computation and recover the performance a larger model trained from scratch with over 50% reduction in computation. |