Papers by Bokai Yu
SpiRit-LM: Interleaved Spoken and Written Language Model (2025.tacl-1)
Copied to clipboard
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, Emmanuel Dupoux
| Challenge: | SpiRit-LM is a foundation multimodal language model that freely mixes text and speech. |
| Approach: | They propose a multimodal language model that freely mixes text and speech . they extend the model to the speech modality by continuously training it on text and language units. |
| Outcome: | The proposed model can learn new tasks in a few-shot fashion across modalities. |
Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing spoken dialogue models are half-duplex in nature and require explicit prompting by the user or implicit tracking of interruption or silence events. |
| Approach: | They propose to integrate time information into Llama3-8b so that they run synchronously with the real-world clock. |
| Outcome: | The proposed model outperforms state-of-the-art in dialogue meaningfulness while maintaining naturalness. |
2M-BELEBELE: Highly Multilingual Speech and American Sign Language Comprehension Dataset Download PDF (2025.findings-acl)
Copied to clipboard
Marta R. Costa-jussà, Bokai Yu, Pierre Andrews, Belen Alastruey, Necati Cihan Camgoz, Joe Chuang, Jean Maillard, Christophe Ropers, Arina Turkatenko, Carleigh Wood
| Challenge: | We extend the BELEBELE dataset to speech and sign, and extend the Automatic Speech Recognition Benchmark, FLEURS, by 20%. |
| Approach: | They extend the BELEBELE and FLEURS speech comprehension datasets to speech and sign . they evaluate the datasets for 5-shot and zero-shot settings and find that the accuracy is 10% lower than reading comprehension. |
| Outcome: | The proposed dataset covers 91 spoken languages and one sign language (ASL) it also extends the Automatic Speech Recognition Benchmark, FLEURS, by 20% across languages. |