Papers by Florian Mai
HyperMixer: An MLP-based Low Cost Alternative to Transformers (2023.acl-long)
Copied to clipboard
Florian Mai, Arnaud Pannatier, Fabio Fehr, Haolin Chen, Francois Marelli, Francois Fleuret, James Henderson
| Challenge: | Existing MLP-based architectures that combine multiple features are expensive and require a lot of training data. |
| Approach: | They propose a simple MLP-based model which allows token mixing by dynamically applying hypernetworks to each feature independently. |
| Outcome: | The proposed model performs better than Transformers and lowers costs in terms of processing time, training data, and hyperparameter tuning. |
Triple-Encoders: Representations That Fire Together, Wire Together (2024.acl-long)
Copied to clipboard
| Challenge: | Existing search-based dialog models re-encode the dialog history at every turn, causing high cost. |
| Approach: | They propose a method that encodes relative distances between utterances via a bi-encoder . they use a hebbian-inspired co-occurrence learning objective to compute distributed utterrance mixtures . |
| Outcome: | The proposed method improves on bi-encoded utterances and even better generalization . it can encode 2020 and Erker et al. (2023) with the same results. |
Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing methods to learn mappings in the embedding space of text autoencoders are limited to a single-vector embeddment, which limits how much information can be retained. |
| Approach: | They propose a method to learn mappings in the embedding space of an autoencoder by extending it to Bag-of-Vectors Autoencodeurs (BoV-AEs) this allows to encode and reconstruct much longer texts than standard autoencodings . |
| Outcome: | The proposed method performs better than a standard autoencoder on unsupervised sentiment transfer. |
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models (2025.emnlp-main)
Copied to clipboard
Mehdi Ali, Manuel Brack, Max Lübbering, Elias Wendt, Abbas Goher Khan, Richard Rutmann, Alex Jude, Maurice Kraus, Alexander Arno Weber, Felix Stollenwerk, David Kaczér, Florian Mai, Lucie Flek, Rafet Sifa, Nicolas Flores-Herr, Joachim Koehler, Patrick Schramowski, Michael Fromm, Kristian Kersting
| Challenge: | Existing open-source multilingual datasets rely on heuristic filtering methods restricting both their cross-lingual transferability and scalability. |
| Approach: | They propose a systematic approach that curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. |
| Outcome: | Evaluated empirically across 35 languages, the proposed approach outperforms current heuristic filtering methods like Fineweb2 and improves model training quality and retention rates. |
Plug and Play Autoencoders for Conditional Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Text autoencoders are used for conditional generation tasks such as style transfer. |
| Approach: | They propose a plug-and-play method where any pretrained autoencoder can be used and only requires learning a mapping within the embedding space. |
| Outcome: | The proposed method performs better than or comparable to strong baselines while being up to four times faster. |