Papers by Benjamin Peloquin
Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent expressive speech-to-speech translation systems have achieved impressive expressivity preservation performances by cascading unit-to speech (U2S) generator to the speech- to-unit translation model. |
| Approach: | They propose a textless acoustic model with a self-supervised distillation strategy for noise-robust expressive speech-to-speech translation (S2ST) They aim to address this limitation by incorporating a distillation with no label (DINO) self-controlled training strategy into the model’s pretraining process. |
| Outcome: | The proposed model significantly improved the expressive speech-to-speech translation system in noisy environments while maintaining competitive performance in clean environments. |
Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing spoken dialogue models are half-duplex in nature and require explicit prompting by the user or implicit tracking of interruption or silence events. |
| Approach: | They propose to integrate time information into Llama3-8b so that they run synchronously with the real-world clock. |
| Outcome: | The proposed model outperforms state-of-the-art in dialogue meaningfulness while maintaining naturalness. |