Papers with XLM
Multilingual Language Models Predict Human Reading Behavior (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that cognitively motivated "attention" mechanism in neural models is not a good indicator for relative importance. |
| Approach: | They compare the performance of language-specific and multilingual pretrained transformer models to predict reading time measures reflecting natural human sentence processing. |
| Outcome: | The proposed models predict reading time measures on Dutch, English, German, and Russian texts. |
Multilingual BERT Post-Pretraining Alignment (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work improves on the success of monolingual pretrained language models by adding cross-lingual tasks that always involve English. |
| Approach: | They propose a method to align multilingual contextual embeddings as a post-pretraining step for improved cross-lingual transferability of pretrained language models. |
| Outcome: | The proposed model outperforms XLM-R_Base on translation-train tasks while using less parallel data and fewer parameters. |
Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing models of multilingual sentence embeddings require large parallel data resources which are not available for low-resource languages. |
| Approach: | They propose an unsupervised method to derive multilingual sentence embeddings using monolingual data. |
| Outcome: | The proposed method improves on two parallel corpus mining tasks and for other languages. |
NepBERTa: Nepali Language Model Trained in a Large Corpus (2022.aacl-short)
Copied to clipboard
| Challenge: | Nepali is a low-resource language with more than 40 million speakers worldwide. |
| Approach: | They present a BERT-based natural language understanding model trained on the most extensive monolingual Nepali corpus ever. |
| Outcome: | The proposed model performs well in Nepali-specific NLP tasks including Named-Entity Recognition, Content Classification, POS Tagging, and Sequence Pair Similarity. |
Multi-Granularity Contrasting for Cross-Lingual Pre-Training (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to pre-training focus on embedding alignment, but they neglect the modeling of bidirectional contexts. |
| Approach: | They propose a framework to learn languageuniversal representations using multi-granularity contrasting framework . they encode semantic equivalents from different languages into similar representations . |
| Outcome: | The proposed framework can achieve significant performance gains in machine translation and cross-lingual language understanding. |
TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in generative language models have enabled machines to generate realistic texts. |
| Approach: | They propose a benchmark environment to test the 'Turing Test' problem for neural text generation methods. |
| Outcome: | The proposed benchmark environment is based on 200K human- or machine-generated samples across 20 labels Human, GPT-1, GTP-2_small, GTT-2_medium, GPG-2_large, GGT-2_PyTorch, GGP-3, GROVER_base, griover_large and GRover_mega. |
Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) models with limited data are ineffective when the two languages are not available for one language. |
| Approach: | They propose an approach that reuses a language model that is pretrained on two languages with large monolingual data to initialize an unsupervised neural machine translation system. |
| Outcome: | The proposed method outperforms a competitive cross-lingual pretraining model in English-Macedonian (En-Mk) and English-Albanian (En Sq) it yields more than +8.3 BLEU points for all four translation directions. |
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks (D19-1)
Copied to clipboard
| Challenge: | Existing models that can handle cross-lingual tasks with limited or no training data are insensitive to different languages. |
| Approach: | They propose to use Unicoder to train models in one language and apply it to other languages. |
| Outcome: | Experiments show that Unicoder learns the mappings among different languages from more perspectives. |
SLING: Sino Linguistic Evaluation of Large Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using pre-trained language models, we find that the accuracy of LMs is far below human performance. |
| Approach: | They propose a benchmark of Sino LINGuistics which consists of 38K sentence pairs in Mandarin Chinese grouped into 9 high-level linguistic phenomena. |
| Outcome: | The proposed model performs better on local phenomena than hierarchical models and has a strong gender and number bias. |
Automatic Machine Translation Evaluation using Source Language Inputs and Cross-lingual Language Model (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for machine translation evaluation use source sentences as pseudo references instead of word symbols. |
| Approach: | They propose an automatic machine translation evaluation method that uses source sentences as pseudo references instead of source sentences. |
| Outcome: | The proposed method achieves higher correlation with human judgments than baseline evaluation method that uses only hypothesis and reference sentences. |
Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference (2021.findings-acl)
Copied to clipboard
| Challenge: | Multilingual transformers have been shown to have remarkable transfer skills in zero-shot settings. |
| Approach: | They investigate cross-lingual transfer abilities of XLM-R for Chinese and English natural language inference using a large scale Chinese dataset. |
| Outcome: | The proposed model trains on Chinese and English natural language inference datasets. |
Intermediate Self-supervised Learning for Machine Translation Quality Estimation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for machine translation quality estimation (QE) rely on annotated data. |
| Approach: | They propose a self-supervised learning task for machine translation (MT) that orients a pre-trained model towards the target task. |
| Outcome: | The proposed method outperforms existing methods on English-to-German and English- to-Russian translation directions and is comparable to existing models. |
Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that backtranslation improves UNMT performance by reducing the data gap between training and inference. |
| Approach: | They propose an online method to remedy the source discrepancy between training and inference . they use pseudo parallel data with translated source and translated target to mimic inference scenario . |
| Outcome: | The proposed method outperforms baselines on several widely-used language pairs by remedying the style and content gaps. |
Exploring Methods for Building Dialects-Mandarin Code-Mixing Corpora: A Case Study in Taiwanese Hokkien (2022.findings-emnlp)
Copied to clipboard
| Challenge: | CM is a challenging task when mixed languages include dialects. |
| Approach: | They propose to construct a Hokkien-Mandarin CM dataset to overcome the limitation . they propose to use a linguistics-based toolkit to train the model for translation tasks . |
| Outcome: | The proposed model achieves good results on CM data translation while maintaining monolingual translation quality. |
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation (2020.emnlp-main)
Copied to clipboard
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, Ming Zhou
| Challenge: | XGLUE provides a benchmark dataset to train large-scale cross-lingual pre-trained models . XCLUE provides 11 diversified tasks that cover both understanding and generation scenarios . |
| Approach: | They introduce a new benchmark dataset to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora. |
| Outcome: | The proposed dataset is labeled in English and includes only natural language understanding tasks. |
Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to learn cross-lingual word embeddings in a contextual space are lacking. |
| Approach: | They propose a method to generate cross-lingual contextualized word embeddings using pre-trained BERT models by learning a linear transformation from contextual word alignments. |
| Outcome: | The proposed approach outperforms state-of-the-art models on zero-shot cross-lingual transfer parsing and is highly competitive with existing models. |
Authorship Attribution for Neural Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep learning have enabled the generation of realistic artifacts . however, the qualities of texts generated by these models are better, often confusing classifiers if they are not real. |
| Approach: | They propose to use neural network-based language models to generate realistic texts . they investigate the authorship attribution problem in three versions of a text . |
| Outcome: | The proposed models generate texts that are difficult to distinguish from human-written ones . the results show that most generators still generate texts significantly different from human ones compared to other models . |
Unsupervised Cross-lingual Representation Learning at Scale (2020.acl-main)
Copied to clipboard
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Pretraining multilingual language models at scale leads to performance gains for cross-lingual transfer tasks. |
| Approach: | They present a transformer-based multilingual masked language model pre-trained on 100 languages . they show that pretraining multilingual models at scale leads to significant performance gains . |
| Outcome: | The proposed model outperforms multilingual BERT (mBERT) on cross-lingual benchmarks. |
Macedon: Minimizing Representation Coding Rate Reduction for Cross-Lingual Natural Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to learn cross-lingual models require limited data to perform cross-linguistic tasks. |
| Approach: | They propose a method to remove language-associated information via minimizing representation coding rate reduction. |
| Outcome: | The proposed model outperforms state-of-the-art models on cross-lingual tasks. |