Papers by Björn Deiseroth
Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation (2026.eacl-long)
Copied to clipboard
Thomas F Burns, Letitia Parcalabescu, Stephan Waeldchen, Michael Barlow, Gregor Ziegltrum, Volker Stampa, Bastian Harren, Björn Deiseroth
| Challenge: | Recent studies show that data quality can significantly boost performance and training efficiency for large language models. |
| Approach: | They propose a German-language dataset curation pipeline that combines heuristic and model-based filtering techniques with synthetic data generation. |
| Outcome: | The proposed pipeline can be used to create a large-scale German pre-training dataset using common Crawl web data, fineweb2 and synthetically generated data conditioned on real, organic web data. |
Divergent Token Metrics: Measuring degradation to prune away LLM components – and optimize quantization (2024.naacl-long)
Copied to clipboard
Björn Deiseroth, Max Meuer, Nikolas Gritsch, Constantin Eichenberg, Patrick Schramowski, Matthias Aßenmacher, Kristian Kersting
| Challenge: | Large Language Models (LLMs) have reshaped natural language processing with impressive capabilities, but their ever-increasing size has raised concerns about their effective deployment and the need for LLM compression. |
| Approach: | This study introduces the Divergent Token Metrics (DTMs) that measure token divergences that allow deeper insights into the subtleties of model compression. |
| Outcome: | The proposed measures can identify outliers and improve performance in the sparseness of the LLMs. |
T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings (2024.emnlp-main)
Copied to clipboard
| Challenge: | Tokenizers are crucial for encoding information in Large Language Models, but their development has stagnated. |
| Approach: | They propose a tokenizer that embeds words through sparse activation patterns over character triplets . they show competitive downstream performance with a parameter reduction of more than 85% . |
| Outcome: | The proposed approach achieves competitive downstream performance with a parameter reduction of more than 85% on embedding layers. |