Papers by Adam Polyak
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021.emnlp-demo)
Copied to clipboard
| Challenge: | Speech synthesis is the task of generating speech waveforms with desired characteristics, including but not limited to textual content, speaker identity, and speaking styles. |
| Approach: | They propose a fairseq extension for speech synthesis that implements autoregressive and non-AR text-to-speech models and their multi-speaker variants. |
| Outcome: | The proposed extension can train autoregressive and non-AR models and their multi-speaker variants with less curated data and has automatic metrics to facilitate faster iteration and analysis. |
Text-Free Prosody-Aware Generative Spoken Language Modeling (2022.acl-long)
Copied to clipboard
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, Wei-Ning Hsu
| Challenge: | Experimental results show that generative spoken language models (LMs) are natural unsupervised multitask learners. |
| Approach: | They propose a prosody-aware generative spoken language model that uses discovered units to generate natural, meaningful, and coherent speech. |
| Outcome: | The proposed model can generate natural, meaningful, and coherent speech given a spoken prompt. |
Direct Speech-to-Speech Translation With Discrete Units (2022.acl-long)
Copied to clipboard
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, Wei-Ning Hsu
| Challenge: | Existing direct speech-to-speech translation models rely on text generation as an intermediate step. |
| Approach: | They propose a direct speech-to-speech translation model that translates speech from one language to another without relying on intermediate text generation. |
| Outcome: | The proposed model produces 6.7 BLEUs in the Fisher Spanish-English dataset when trained without any text transcripts and with text supervision. |
Textless Speech Emotion Conversion using Discrete & Decomposed Representations (2022.emnlp-main)
Copied to clipboard
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu Anh Nguyen, Morgan Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, Yossi Adi
| Challenge: | Existing methods for modifying emotion of speech are difficult because emotion affects all levels simultaneously. |
| Approach: | They propose a method to convert a spoken language speech into a model of emotion . they use phonetic-content units, prosodic features, speaker, and emotion to modify the emotion a speech utterance has. |
| Outcome: | The proposed method beats text-based systems in terms of perceived emotion and audio quality. |
On Generative Spoken Language Modeling from Raw Audio (2021.tacl-1)
Copied to clipboard
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, Emmanuel Dupoux
| Challenge: | Using a set of metrics to evaluate the learned representations, we aim to create a system that learns from natural interactions as infants learn their first language. |
| Approach: | They propose a task of learning acoustic and linguistic characteristics from raw audio and a set of metrics to evaluate the learned representations at acustic, linguistic and encoding levels. |
| Outcome: | The proposed models evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. |