Papers by Constantin Eichenberg
Divergent Token Metrics: Measuring degradation to prune away LLM components – and optimize quantization (2024.naacl-long)
Copied to clipboard
Björn Deiseroth, Max Meuer, Nikolas Gritsch, Constantin Eichenberg, Patrick Schramowski, Matthias Aßenmacher, Kristian Kersting
| Challenge: | Large Language Models (LLMs) have reshaped natural language processing with impressive capabilities, but their ever-increasing size has raised concerns about their effective deployment and the need for LLM compression. |
| Approach: | This study introduces the Divergent Token Metrics (DTMs) that measure token divergences that allow deeper insights into the subtleties of model compression. |
| Outcome: | The proposed measures can identify outliers and improve performance in the sparseness of the LLMs. |
MAGMA – Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pretraining is becoming the norm in Vision-Language (VL) modeling. |
| Approach: | They propose a method for augmenting generative language models with additional modalities using adapter-based finetuning. |
| Outcome: | The proposed method outperforms Frozen on open-ended generative tasks while maintaining the language model weights. |