Papers by Magnus Sahlgren
Lessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedish (2022.lrec-1)
Copied to clipboard
Ariel Ekgren, Amaru Cuba Gyllensten, Evangelia Gogoulou, Alice Heiman, Severine Verlinden, Joey Öhman, Fredrik Carlsson, Magnus Sahlgren
| Challenge: | a prerequisite for building large-scale generative models for other languages is access to large amounts of high-quality text data and powerful computational resources. |
| Approach: | They present a 3.5 billion parameter autoregressive language model, trained on a 100 GB Swedish corpus. |
| Outcome: | The proposed model performs well on a 100 GB Swedish corpus and is competent in comparison with existing models of similar size. |
Superlim: A Swedish Language Understanding Evaluation Benchmark (2023.emnlp-main)
Copied to clipboard
Aleksandrs Berdicevskis, Gerlof Bouma, Robin Kurtz, Felix Morger, Joey Öhman, Yvonne Adesam, Lars Borin, Dana Dannélls, Markus Forsberg, Tim Isbister, Anna Lindahl, Martin Malmsten, Faton Rekathati, Magnus Sahlgren, Elena Volodina, Love Börjeson, Simon Hengchen, Nina Tahmasebi
| Challenge: | In this paper, we present a multi-task benchmark for Swedish language models . we address methodological challenges, such as mitigating the Anglocentric bias when creating datasets for a less-resourced language . |
| Approach: | They propose a multi-task NLP benchmark for Swedish language models . they propose to use superlim to evaluate Swedish language model performance . |
| Outcome: | The proposed benchmark does not approach ceiling performance on any of the tasks, suggesting it is difficult to implement. |
Distributional Term Set Expansion (L18-1)
Copied to clipboard
| Challenge: | Iterative term set expansion methods for distributional semantic models are used to label terms belonging to a sought after term set. |
| Approach: | They compare iterative term set expansion methods for distributional semantic models to the Simple Margin method, an active learning approach to classification using Support Vector Machines. |
| Outcome: | The proposed methods outperform centrality and classification based methods for distributional semantic models over five different term sets. |
Cross-lingual and Multilingual CLIP (2022.lrec-1)
Copied to clipboard
| Challenge: | OpenAI released CLIP, a model that relates the textual and visual domains with unprecedented accuracy. |
| Approach: | They propose to use cross-lingual teacher learning to re-train an English textual encoder using a large dataset of images and captions. |
| Outcome: | The proposed method outperforms baselines on multilingual image-text retrieval while retaining low cost. |
Predicting Treatment Outcome from Patient Texts:The Case of Internet-Based Cognitive Behavioural Therapy (2021.eacl-main)
Copied to clipboard
Evangelia Gogoulou, Magnus Boman, Fehmi Ben Abdesslem, Nils Hentati Isacsson, Viktor Kaldo, Magnus Sahlgren
| Challenge: | Existing methods to predict treatment outcome are limited to text categorisation, but they can be applied to patient texts. |
| Approach: | They propose to use patient text as the only signal for predicting treatment outcome in Internet-based cognitive behavioural therapy for depression, social anxiety, and panic disorder. |
| Outcome: | The proposed method beats stratified random guessing by using a simple Bag of Words to predict treatment success and failure. |
Fine-Grained Controllable Text Generation Using Non-Residual Prompting (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to control the text generation process are not expressive enough. |
| Approach: | They propose an encoder-decoder architecture that enables intermediate text prompts at arbitrary time steps. |
| Outcome: | The proposed architecture is expressive and versatile on multiple experimental settings. |
GPT-SW3: An Autoregressive Language Model for the Scandinavian Languages (2024.lrec-main)
Copied to clipboard
Ariel Ekgren, Amaru Cuba Gyllensten, Felix Stollenwerk, Joey Öhman, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, Judit Casademont, Magnus Sahlgren
| Challenge: | a growing interest in building and applying large language models for languages other than English is fueling interest in developing LLMs for smaller languages. |
| Approach: | They describe the development process for the first native large generative language model for the North Germanic languages, GPT-SW3. |
| Outcome: | The proposed model is based on the generative language model for the North Germanic languages . it is a first-generation model with a high-quality data set and a low cost of implementation . |
Cross-lingual Transfer of Monolingual Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual learning using multilingual models cast doubt on shared vocabulary and joint pre-training . et al. (2005) show that model knowledge learned in the source language enhances the learning of the target language independently of language proximity. |
| Approach: | They propose a method for transferring monolingual models to other languages through continuous pre-training and investigate their results in English. |
| Outcome: | The proposed method outperforms a model trained from scratch in the GLUE benchmark for English . it shows that model knowledge from the source language enhances the learning of syntactic and semantic knowledge in english. |
Rethinking Topic Modelling: From Document-Space to Term-Space (2020.findings-emnlp)
Copied to clipboard
| Challenge: | a paper examines the use of document-based topic models for defining term interactions . term-based models outperform document-derived models by a large margin, the authors argue . |
| Approach: | They propose to reformulate topic distributions as latent factors in term similarity space . they use word embeddings built with wide context windows to test their approach . |
| Outcome: | The proposed model outperforms standard document-based models by a large margin . the proposed model is less sensitive to document formatting and more precise . |