Papers by Acyr Locatelli
Nexus: Adaptive Upcycling to Efficiently Pretrain Mixture of Experts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Nevertheless, training from scratch on trillions of tokens remains expensive that most users can only finetune these models. |
| Approach: | They propose to reuse parameters of dense models for the MoE layers with a router that can integrate new experts into existing trained models without hurting performance on previous domains. |
| Outcome: | The proposed router can integrate new experts into existing trained models without hurting the performance on previous domains. |
One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers (2026.acl-long)
Copied to clipboard
Diana Abagyan, Alejandro R. Salamanca, Andres Felipe Cruz-Salinas, Kris Cao, Hangyu Lin, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, Sara Hooker
| Challenge: | Existing approaches to train multilingual large language models for many languages at once are limited due to limited model capacity, scarce high-quality data, and compute constraints. |
| Approach: | They propose to use a universal tokenizer to improve language plasticity and adaptability to new languages by up to 20%. |
| Outcome: | The proposed tokenizer improves language plasticity and improves plasticity towards languages that are completely unseen in the tokenizer and pretraining, by up to 5% win rate gain. |