Papers by Saksham Singhal
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2022.acl-long)
Copied to clipboard
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, Furu Wei
| Challenge: | ELECTRA-style tasks are used to pretrain cross-lingual models for NLP tasks . masked language modeling tasks require massive computation resources, rendering such models quite expensive . |
| Approach: | They propose to use ELECTRA-style tasks to pre-train a cross-lingual language model . they propose to pretrain the model on multilingual and parallel corpora . |
| Outcome: | The proposed model outperforms baseline models on cross-lingual understanding tasks with much less computation cost. |
Allocating Large Vocabulary Capacity for Cross-Lingual Language Model Pre-Training (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models require a more expressive vocabulary to represent all languages . however, increasing the vocabulary size significantly slows down the pre-training speed . |
| Approach: | They propose an algorithm VoCap to determine the desired vocabulary capacity of each language. |
| Outcome: | The proposed algorithm improves cross-lingual model pre-training while reducing side effects of increasing vocabulary size. |
mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021.emnlp-main)
Copied to clipboard
Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Saksham Singhal, Xian-Ling Mao, Heyan Huang, Xia Song, Furu Wei
| Challenge: | Multilingual T5 pretrains a sequence-to-sequence model on monolingual texts, but it has shown promising results on many cross-lingual tasks. |
| Approach: | They propose a partially non-autoregressive objective for text-to-text pre-training and propose mT6 to improve cross-lingual transferability over multilingual T5. |
| Outcome: | The proposed model improves cross-lingual transferability over existing models. |
On the Adaptation of Unlimiformer for Decoder-Only Transformers (2024.lrec-main)
Copied to clipboard
| Challenge: | despite efforts in the community, most common models have a context length of 4k or less. |
| Approach: | They propose to adapt a vector-retrieval augmentation method to decoder-only transformers . they also expand the experimental setup on summarization to include a new task and an instruction-tuned model . |
| Outcome: | The proposed model performs on par with a model with 2x the context length. |
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning (2023.acl-long)
Copied to clipboard
Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
| Challenge: | XY-LENT: X-Y bitext enhanced Language ENcodings achieves state-of-the-art performance over 5 cross-lingual tasks within all model size bands. |
| Approach: | They propose a method for building multilingual representation models that are competitive with existing models and more parameter efficient. |
| Outcome: | The proposed model outperforms XLM-R XXL and is 5x and 6x smaller respectively. |
Consistency Regularization for Cross-Lingual Fine-Tuning (2021.acl-long)
Copied to clipboard
Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
| Challenge: | Experimental results show that consistency regularization improves cross-lingual fine-tuning . pre-trained cross-linguistic models can transfer task-specific supervision from one language to the other . |
| Approach: | They propose to improve cross-lingual fine-tuning with consistency regularization . they use example consistency regularized to penalize prediction sensitivity to four types of data augmentations . |
| Outcome: | The proposed method improves cross-lingual fine-tuning across tasks . it can be generalized to other target languages without additional training . |
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2021.naacl-main)
Copied to clipboard
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
| Challenge: | Existing methods for learning cross-lingual representations are lacking in the field of NLP. |
| Approach: | They propose a framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. |
| Outcome: | The proposed approach improves cross-lingual transferability on benchmarks. |