Papers by Ariel Ekgren
Lessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedish (2022.lrec-1)
Copied to clipboard
Ariel Ekgren, Amaru Cuba Gyllensten, Evangelia Gogoulou, Alice Heiman, Severine Verlinden, Joey Öhman, Fredrik Carlsson, Magnus Sahlgren
| Challenge: | a prerequisite for building large-scale generative models for other languages is access to large amounts of high-quality text data and powerful computational resources. |
| Approach: | They present a 3.5 billion parameter autoregressive language model, trained on a 100 GB Swedish corpus. |
| Outcome: | The proposed model performs well on a 100 GB Swedish corpus and is competent in comparison with existing models of similar size. |
GPT-SW3: An Autoregressive Language Model for the Scandinavian Languages (2024.lrec-main)
Copied to clipboard
Ariel Ekgren, Amaru Cuba Gyllensten, Felix Stollenwerk, Joey Öhman, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, Judit Casademont, Magnus Sahlgren
| Challenge: | a growing interest in building and applying large language models for languages other than English is fueling interest in developing LLMs for smaller languages. |
| Approach: | They describe the development process for the first native large generative language model for the North Germanic languages, GPT-SW3. |
| Outcome: | The proposed model is based on the generative language model for the North Germanic languages . it is a first-generation model with a high-quality data set and a low cost of implementation . |
Cross-lingual Transfer of Monolingual Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual learning using multilingual models cast doubt on shared vocabulary and joint pre-training . et al. (2005) show that model knowledge learned in the source language enhances the learning of the target language independently of language proximity. |
| Approach: | They propose a method for transferring monolingual models to other languages through continuous pre-training and investigate their results in English. |
| Outcome: | The proposed method outperforms a model trained from scratch in the GLUE benchmark for English . it shows that model knowledge from the source language enhances the learning of syntactic and semantic knowledge in english. |