Papers by Felix Stollenwerk
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models (2025.emnlp-main)
Copied to clipboard
Mehdi Ali, Manuel Brack, Max Lübbering, Elias Wendt, Abbas Goher Khan, Richard Rutmann, Alex Jude, Maurice Kraus, Alexander Arno Weber, Felix Stollenwerk, David Kaczér, Florian Mai, Lucie Flek, Rafet Sifa, Nicolas Flores-Herr, Joachim Koehler, Patrick Schramowski, Michael Fromm, Kristian Kersting
| Challenge: | Existing open-source multilingual datasets rely on heuristic filtering methods restricting both their cross-lingual transferability and scalability. |
| Approach: | They propose a systematic approach that curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. |
| Outcome: | Evaluated empirically across 35 languages, the proposed approach outperforms current heuristic filtering methods like Fineweb2 and improves model training quality and retention rates. |
GPT-SW3: An Autoregressive Language Model for the Scandinavian Languages (2024.lrec-main)
Copied to clipboard
Ariel Ekgren, Amaru Cuba Gyllensten, Felix Stollenwerk, Joey Öhman, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, Judit Casademont, Magnus Sahlgren
| Challenge: | a growing interest in building and applying large language models for languages other than English is fueling interest in developing LLMs for smaller languages. |
| Approach: | They describe the development process for the first native large generative language model for the North Germanic languages, GPT-SW3. |
| Outcome: | The proposed model is based on the generative language model for the North Germanic languages . it is a first-generation model with a high-quality data set and a low cost of implementation . |
On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions (2026.eacl-short)
Copied to clipboard
| Challenge: | Layer normalization (LN) is an essential component of modern neural networks. |
| Approach: | They propose a dynamic activation function called Dynamic Tanh which is based on the LN variant RMSNorm and decouples in derivative space. |
| Outcome: | The proposed function reproduces the normalization effect on outliers more accurately than RMSNorm by using a well-defined decoupling procedure in derivative space and an approximation. |
Better Embeddings with Coupled Adam (2025.acl-long)
Copied to clipboard
| Challenge: | Anisotropic Embeddings Large Language Models exhibit undesirable yet poorly understood feature of anisotropy. |
| Approach: | They propose an algorithm that uses the second moment in Adam to mitigate anisotropic embeddings . they propose an embeddable matrix and unembedding matrix to map the input and output tokens based on weight tying . |
| Outcome: | The proposed model improves quality and performance on large datasets. |