Papers by Martin Jaggi
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (N18-1)
Copied to clipboard
| Challenge: | Currently, unsupervised word embeddings are routinely trained on large amounts of raw text data. |
| Approach: | They propose to use unsupervised word embeddings to train distributed representations of sentences. |
| Outcome: | The proposed method outperforms state-of-the-art models on most benchmark tasks and is robust to the produced general-purpose sentence embeddings. |
SKILL: Structured Knowledge Infusion for Large Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated human-level performance on a vast spectrum of natural language tasks. |
| Approach: | They propose a method to infuse structured knowledge into large language models by directly training T5 models on factual triples of knowledge graphs (KGs). |
| Outcome: | The proposed method outperforms baseline models on FreebaseQA and WikiHop, as well as the Wikidata-answerable subset of TriviaQA and NaturalQuestions. |
Better Word Embeddings by Disentangling Contextual n-Gram Information (N19-1)
Copied to clipboard
| Challenge: | Pre-trained word vectors are ubiquitous in Natural Language Processing applications. |
| Approach: | They show that word embeddings with bigram and trigram embedds improve unigram embeds . they claim this removes contextual information from unigrammes, resulting in better unigraph embedders . |
| Outcome: | The proposed model outperforms competing models on a wide variety of tasks. |
Self-Supervised Neural Topic Modeling (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Topic models are useful tools for analyzing and interpreting the main underlying themes of large corpora of text. |
| Approach: | They propose a self-supervised neural topic model that learns a topic representation jointly from three co-occurring words and a document that the triple originates from. |
| Outcome: | The proposed model outperforms existing topic models in coherence metrics and document clustering accuracy. |
Lightweight Cross-Lingual Sentence Representation Learning (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for learning fixed-dimensional cross-lingual sentence representations are impractical due to memory limitations. |
| Approach: | They propose a lightweight dual-transformer architecture with just 2 layers for generating memory-efficient cross-lingual sentence representations. |
| Outcome: | The proposed model improves performance on training tasks and improves memory efficiency. |
SIMSUM: Document-level Text Simplification via Simultaneous Summarization (2023.acl-long)
Copied to clipboard
| Challenge: | Document-level text simplification is a specific type of simplification which involves simplifying documents consisting of several sentences by rewriting them into fewer or more sentences. |
| Approach: | They propose a new two-stage framework SIMSUM for automated document-level text simplification which uses explicit summarization and simplification models and guides the generation using the main keywords of a source text. |
| Outcome: | The proposed model outperforms baseline models on two document-level simplification datasets, namely D-Wikipedia and Wiki-Doc. |
Obtaining Better Static Word Embeddings Using Contextual Embedding Models (2021.acl-long)
Copied to clipboard
| Challenge: | Recent contextual word embeddings have prohibitively high computational cost in many use-cases and are hard to interpret. |
| Approach: | They propose a distillation method which is an extension of CBOW-based training and improves computational efficiency of NLP applications. |
| Outcome: | The proposed method outperforms existing models and existing models in terms of quality and performance. |
Apertus: Democratizing Open and Compliant LLMs for Global Language Environments (2026.acl-long)
Copied to clipboard
Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pásztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, Ido Hakimi, Juan Garcia Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, Inés Altemir Marinas, Mohammad Hossein Amani, Matin Ansaripour, Ilia Badanin, Harold Benoit, Emanuela Boros, Nicholas John Browning, Fabian Bösch, Maximilian Böther, Niklas Canova, Camille Challier, Clément Charmillot, Jonathan Coles, Jan Milan Deriu, Arnout Devos, Lukas Drescher, Daniil Dzenhaliou, Maud Ehrmann, Dongyang Fan, Simin Fan, Silin Gao, Miguel Gila, María Grandury, Diba Hashemi, Alexander Miserlis Hoyle, Jiaming Jiang, Mark Klein, Andrei Kucharavy, Anastasiia Kucherenko, Frederike Lübeck, Roman Machacek, Theofilos Ioannis Manitaras, Andreas Marfurt, Kyle Matoba, Simon Matrenok, Henrique Mendonça, Fawzi Roberto Mohamed, Syrielle Montariol, Luca Mouchel, Sven Najem-Meyer, Jingwei Ni, Gennaro Oliva, Matteo Pagliardini, Elia Palme, Andrei Panferov, Léo Paoletti, Marco Passerini, Ivan Pavlov, Auguste Poiroux, Kaustubh Ponkshe, Nathan Ranchin, Javier Rando, Mathieu Sauser, Jakhongir Saydaliev, Mukhammadali Sayfiddinov, Marian Schneider, Stefano Schuppli, Marco Scialanga, Andrei Semenov, Kumar Shridhar, Raghav Singhal, Anna Sotnikova, Alexander Sternfeld, Ayush Kumar Tarun, Paul Teiletche, Jannis Vamvas, Xiaozhe Yao, Hao Zhao, Alexander Ilic, Ana Klimovic, Andreas Krause, Caglar Gulcehre, David Rosenthal, Elliott Ash, Florian Tramèr, Joost VandeVondele, Livio Veraldi, Martin Rajman, Thomas C. Schulthess, Torsten Hoefler, Antoine Bosselut, Martin Jaggi, Imanol Schlag
| Challenge: | Apertus is a fully open suite of large language models (LLMs) designed to address responsibility shortcomings in today’s open model ecosystem, namely data responsibility and global representation. |
| Approach: | They propose to release a fully open suite of large language models (LLMs) that address data responsibility and global representation shortcomings in today’s open model ecosystem. |
| Outcome: | The proposed model is pretrained on openly available data and suppresses verbatim recall of data while retaining task performance. |
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that our binary masked language models encode information necessary for solving downstream tasks. |
| Approach: | They propose an efficient method of utilizing pretrained language models where selective binary masks are learned instead of finetuning. |
| Outcome: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that the proposed method yields comparable performance to finetuning, but has a much smaller memory footprint when multiple tasks need to be solved. |