Papers by Linting Xue
MaXM: Towards Multilingual Visual Question Answering (2023.findings-emnlp)
Copied to clipboard
Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, Radu Soricut
| Challenge: | Visual Question Answering (VQA) has been studied in the English language, but in other languages it would require a considerable amount of resources. |
| Approach: | They propose scalable solutions to multilingual visual question answering using an English language framework and an annotation protocol. |
| Outcome: | The proposed framework reduces human annotation efforts and creates a test-only VQA benchmark in 7 languages. |
nmT5 - Is parallel data still relevant for pre-training massively multilingual language models? (2021.acl-short)
Copied to clipboard
| Challenge: | Recent studies have shown that cross-lingual transfer learning in pre-trained multilingual models could be improved further by incorporating parallel data. |
| Approach: | They propose to integrate parallel data into mT5 pre-training to improve results on downstream multilingual and cross-lingual tasks. |
| Outcome: | The proposed model improves cross-lingual transfer significantly in small fine-tuning datasets and small model sizes. |
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2022.tacl-1)
Copied to clipboard
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel
| Challenge: | a number of pre-trained language models use sequences of tokens corresponding to word units . token-free models that operate directly on raw text have many advantages . |
| Approach: | They propose a standard Transformer architecture that can be used to process byte sequences . they also characterize trade-offs in terms of parameter count, training FLOPs, and inference speed . |
| Outcome: | The proposed model is more robust to noise and more robust on spelling and pronunciation tasks. |
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2021.naacl-main)
Copied to clipboard
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel
| Challenge: | Current natural language processing pipelines often use transfer learning, where a model is pre-trained on a data-rich task before being fine-tuned on . this significantly limits their use given that roughly 80% of the world population does not speak English. |
| Approach: | They introduce a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. |
| Outcome: | The proposed model achieves state-of-the-art on multilingual benchmarks and a simple technique to prevent accidental translation in the zero-shot setting. |