Papers with PAWS-X
Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to pretrain Masked Language Models (MLMs) are expensive and require a full forward and backward pass over the entire model. |
| Approach: | They propose to learn a shallow mini-model from a fraction of a large model's parameters and plug it into a larger model for rapid cross-lingual transfer. |
| Outcome: | Experiments on XNLI, MLQA and PAWS-X show that mini-model adaptation matches the standard approach using up to 2.3x less compute on average. |
Syntax-augmented Multilingual BERT for Cross-lingual Transfer (2021.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained multilingual text encoders capture language syntax, helping cross-lingual transfer. |
| Approach: | They provide language syntax and train mBERT to encode universal dependency tree structure. |
| Outcome: | The proposed model improves cross-lingual transfer on PAWS-X and MLQA benchmarks by 1.4 and 1.6 points on average across all languages. |
PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification (D19-1)
Copied to clipboard
| Challenge: | Existing work on adversarial data generation focuses on English . Existing multilingual datasets show effectiveness of deep, multilingual pre-training . |
| Approach: | They propose a dataset of 23,659 human translated PAWS evaluation pairs in six languages . they show the effectiveness of deep, multilingual pre-training while leaving considerable headroom . |
| Outcome: | The proposed model shows that multilingual training and evaluation regimes are more accurate than previous models. |