Papers by Genta Winata
Re-Evaluating Evaluation for Multilingual Summarization (2024.emnlp-main)
Copied to clipboard
Jessica Forde, Ruochen Zhang, Lintang Sutawika, Alham Aji, Samuel Cahyawijaya, Genta Winata, Minghao Wu, Carsten Eickhoff, Stella Biderman, Ellie Pavlick
| Challenge: | Existing studies have shown that automated evaluation approaches correlate with human ratings in English, but this is unclear for other languages. |
| Approach: | They construct a small-scale pilot dataset containing article-summary pairs and human ratings in English, Chinese and Indonesian to measure the strength of summaries. |
| Outcome: | The results show that standard metrics are unreliable measures of quality in Chinese and Indonesian. |
SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages (2024.findings-acl)
Copied to clipboard
Nedjma Ousidhoum, Shamsuddeen Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Ahmad, Sanchit Ahuja, Alham Aji, Vladimir Araujo, Abinew Ayele, Pavan Baswani, Meriem Beloucif, Chris Biemann, Sofia Bourhim, Christine Kock, Genet Dekebo, Oumaima Hourrane, Gopichand Kanumolu, Lokesh Madasu, Samuel Rutunda, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Hailegnaw Tilaye, Krishnapriya Vishnubhotla, Genta Winata, Seid Yimam, Saif Mohammad
| Challenge: | SemRel datasets are annotated by native speakers across 13 languages . they are used to characterise the relationship between two units of text . |
| Approach: | They propose to use a semantic relatedness dataset to measure the degree of semantic textual relatedness between sentences in Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Spanish, and Telugu. |
| Outcome: | The proposed datasets are annotated by native speakers across 13 languages and represent the semantic relatedness of 13 languages. |
Multilingual Large Language Models Are Not (Yet) Code-Switchers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual Large Language Models are not specifically trained with objectives for managing code-switching scenarios. |
| Approach: | They propose to use multilingual Large Language Models to perform sentiment analysis, machine translation, summarization and word-level language identification to compare their performance to fine-tuned models of much smaller scales. |
| Outcome: | The proposed models show that they underperform in comparison to fine-tuned models of much smaller scales. |
NusaCrowd: Open Source Initiative for Indonesian NLP Resources (2023.findings-acl)
Copied to clipboard
Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Winata, Bryan Wilie, Fajri Koto, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, Jennifer Santoso, David Moeljadi, Cahya Wirawan, Frederikus Hudi, Muhammad Satrio Wicaksono, Ivan Parmonangan, Ika Alfina, Ilham Firdausi Putra, Samsul Rahmadani, Yulianti Oenang, Ali Septiandri, James Jaya, Kaustubh Dhole, Arie Suryani, Rifki Afina Putri, Dan Su, Keith Stevens, Made Nindyatama Nityasya, Muhammad Adilazuarda, Ryan Hadiwijaya, Ryandito Diandaru, Tiezheng Yu, Vito Ghifari, Wenliang Dai, Yan Xu, Dyah Damapuspita, Haryo Wibowo, Cuk Tho, Ichwanul Karo Karo, Tirana Fatyanosa, Ziwei Ji, Graham Neubig, Timothy Baldwin, Sebastian Ruder, Pascale Fung, Herry Sujaini, Sakriani Sakti, Ayu Purwarianti
| Challenge: | Existing NLP research in Indonesian languages has been held back by factors such as language diversity, orthographic variation, resource limitation and other societal challenges. |
| Approach: | They present a collaborative initiative to collect and unify existing resources for Indonesian languages and open access to previously non-public resources. |
| Outcome: | The results show that the datasets are highly reliable and can be used to generate the first zero-shot benchmarks for natural language understanding and generation in Indonesian and the local languages of Indonesia. |
Multi-lingual and Multi-cultural Figurative Language Understanding (2023.findings-acl)
Copied to clipboard
Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, Graham Neubig
| Challenge: | Figures permeate human communication, but are understudied in NLP. |
| Approach: | They create a figurative language inference dataset for seven languages associated with a variety of cultures, using cultural and regional concepts for figurativ expressions. |
| Outcome: | The results show that the most common figurative expressions are found in Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili and Yoruba. |
GlobalBench: A Benchmark for Global Progress in Natural Language Processing (2023.emnlp-main)
Copied to clipboard
Yueqi Song, Simran Khanuja, Pengfei Liu, Fahim Faisal, Alissa Ostapenko, Genta Winata, Alham Aji, Samuel Cahyawijaya, Yulia Tsvetkov, Antonios Anastasopoulos, Graham Neubig
| Challenge: | despite advances in NLP, significant disparities in performance across languages still exist . prior benchmarks focused on a limited number of tasks and languages, but now GlobalBench tracks progress on all languages. |
| Approach: | They propose to use global benchmarks to track progress on all NLP datasets in all languages. |
| Outcome: | a new tool tracks progress on all NLP datasets in all languages and tracks per-speaker utility and equity . globalbench is designed to identify the most under-served languages and reward research efforts . a globalbech is available at https://github.com/neulab/globalbench. |
LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models have shown remarkable generalization toward multiple tasks and languages, but their generalization towards unseen languages is poor. |
| Approach: | They propose a regularization technique that incorporates various aspects of languages to better characterize linguistics constraints. |
| Outcome: | The proposed technique improves accuracy of mBERT and XLM-R on unseen languages by 18% and 2% compared to fully finetuned models. |
Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning (2023.findings-acl)
Copied to clipboard
Genta Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, Daniel Preotiuc-Pietro
| Challenge: | Existing methods to handle catastrophic forgetting fail to retain knowledge learnt in the past when sudden shifts occur in training data distributions. |
| Approach: | They propose a learning rate scheduling method that preserves new information without strongly overwriting past knowledge. |
| Outcome: | The proposed method preserves new information without overwriting past knowledge in a multilingual continuous learning framework. |
CI-AVSR: A Cantonese Audio-Visual Speech Datasetfor In-car Command Recognition (2022.lrec-1)
Copied to clipboard
Wenliang Dai, Samuel Cahyawijaya, Tiezheng Yu, Elham J. Barezi, Peng Xu, Cheuk Tung Yiu, Rita Frieske, Holy Lovenia, Genta Winata, Qifeng Chen, Xiaojuan Ma, Bertram Shi, Pascale Fung
| Challenge: | In-car smart assistants should be able to process general as well as car-related commands and perform corresponding actions, which eases driving and improves safety. |
| Approach: | They propose a dataset for in-car command recognition in the cantonese language with both video and audio data. |
| Outcome: | The proposed model can achieve a considerable quality on the clean test set, but the speech recognition quality on noisy data is still inferior. |
ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation (2022.lrec-1)
Copied to clipboard
Holy Lovenia, Samuel Cahyawijaya, Genta Winata, Peng Xu, Yan Xu, Zihan Liu, Rita Frieske, Tiezheng Yu, Wenliang Dai, Elham J. Barezi, Qifeng Chen, Xiaojuan Ma, Bertram Shi, Pascale Fung
| Challenge: | Code-switching is a speech phenomenon occurring when a speaker switches language during a conversation. |
| Approach: | They propose to collect Mandarin Chinese-English code-switching corpus from read speech rather than spontaneous speech to address this phenomenon. |
| Outcome: | ASCEND consists of 10.62 hours of clean speech, collected from 23 bilingual speakers of Chinese and English. |
Academics Can Contribute to Domain-Specialized Language Models (2024.emnlp-main)
Copied to clipboard
Mark Dredze, Genta Winata, Prabhanjan Kambadur, Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, David Rosenberg, Sebastian Gehrmann
| Challenge: | Commercially available models dominate academic leaderboards, focusing on creating and adapting general-purpose models . however, general- purpose models often underperform in specialized domains, and domain-specific models yield superior results. |
| Approach: | They advocate for a renewed focus on developing and evaluating domain- and task-specific models . they advocate for an adapted or adapted model that can be used to improve academic leaderboard standings . |
| Outcome: | The proposed model can do well on professional and linguistic examinations, college-level knowledge questions, and collections of reasoning tasks. |
Cross-lingual Few-Shot Learning on Unseen Languages (2022.aacl-main)
Copied to clipboard
| Challenge: | Large pre-trained language models have demonstrated the ability to obtain good performance on downstream tasks with limited examples in resource-rich languages. |
| Approach: | They propose to use a downstream sentiment analysis task to analyze the effectiveness of several few-shot learning strategies across 12 languages, including 8 unseen languages, to compare results. |
| Outcome: | The proposed model, XLM-R, gives the best performance on a task with few examples in resource-rich languages. |
MINERS: Multilingual Language Models as Semantic Retrievers (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks have evaluated language models to evaluate their performance across a range of embedding tasks. |
| Approach: | They propose a benchmark to evaluate the robustness of multilingual language models in semantic retrieval tasks including bitext mining and classification via retrieval-augmented contexts. |
| Outcome: | The proposed framework evaluates the robustness of multilingual LMs in retrieval tasks across over 200 languages, including extremely low-resource languages in challenging cross-lingual and code-switching settings. |
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2023.acl-long)
Copied to clipboard
Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, Vassilina Nikoulina
| Challenge: | Existing language adaptation strategies for multilingual models are limited to 46 languages . a new language is added to the model to improve zero-shot prompting performance . |
| Approach: | They apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting. |
| Outcome: | The proposed model can be extended to other languages without incurring prohibitively large costs. |
SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages (2024.emnlp-main)
Copied to clipboard
Holy Lovenia, Rahmad Mahendra, Salsabil Akbar, Lester James Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno Kampman, Joel Moniz, Muhammad Habibi, Frederikus Hudi, Jann Montalan, Ryan Hadiwijaya, Joanito Lopo, William Nixon, Börje Karlsson, James Jaya, Ryandito Diandaru, Yuze Gao, Patrick Irawan, Bin Wang, Jan Christian Blaise Cruz, Chenxi Whitehouse, Ivan Parmonangan, Maria Khelli, Wenyu Zhang, Lucky Susanto, Reynard Ryanda, Sonny Hermawan, Dan Velasco, Muhammad Kautsar, Willy Hendria, Yasmin Moslem, Noah Flynn, Muhammad Adilazuarda, Haochen Li, Johanes Lee, R. Damanhuri, Shuo Sun, Muhammad Qorib, Amirbek Djanibekov, Wei Qi Leong, Quyet V. Do, Niklas Muennighoff, Tanrada Pansuwan, Ilham Firdausi Putra, Yan Xu, Tai Chia, Ayu Purwarianti, Sebastian Ruder, William Tjhi, Peerat Limkonchotiwat, Alham Aji, Sedrick Keh, Genta Winata, Ruochen Zhang, Fajri Koto, Zheng Xin Yong, Samuel Cahyawijaya
| Challenge: | Southeast Asia (SEA) is home to over 1,300 indigenous languages and 671 million people . prevailing AI models suffer from a significant lack of representation of texts, images, and audio datasets from SEA . |
| Approach: | They propose to provide a resource center that provides standardized corpora in nearly 1,000 SEA languages across three modalities. |
| Outcome: | a new benchmark assesses the quality of AI models on 36 SEA languages across 13 tasks . the results highlight the importance of SEA as a culturally diverse region . |
The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges (2023.findings-acl)
Copied to clipboard
| Challenge: | Code-Switching is a common phenomenon in written text and conversation . it is not so common to observe code-switching in spoken language and not in written language . |
| Approach: | They present a systematic survey on code-switching research in natural language processing to understand the progress of the past decades and conceptualize the challenges and tasks on the topic. |
| Outcome: | The proposed model combines linguistic theories and machine learning techniques to understand the code-switching phenomenon. |
On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research (2023.acl-long)
Copied to clipboard
Made Nindyatama Nityasya, Haryo Wibowo, Alham Fikri Aji, Genta Winata, Radityo Eko Prasojo, Phil Blunsom, Adhiguna Kuncoro
| Challenge: | Despite rapid recent progress, current research practices conflate different sources of model improvement without conducting proper ablation studies and principled comparisons . authors conclude with recommendations for how to encourage and incentivize this line of work . |
| Approach: | They critique current research practices in the field of language model pre-training . they examine the success of language models pre-trained on large amounts of data . |
| Outcome: | The proposed models can achieve competitive or better performance than BERT under comparable conditions. |