Generative Spoken Language Model based on continuous word-sized audio tokens (2023.emnlp-main)
Copied to clipboard
| Challenge: | Text-based language models outperform character-based models, but speech inputs are 20ms or 40ms-long discrete units. |
| Approach: | They propose a generative language model based on word-size continuous audio tokens . they replace lookup table for lexical types with a Lexical Embedding function . |
| Outcome: | The proposed model is five times more memory efficient than discrete unit GSLMs and is phonetically and semantically interpretable. |
Similar Papers
Evaluating Text-to-Speech Synthesis from a Large Discrete Token-based Speech Language Model (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in generative language modeling applied to discrete speech tokens presented a new avenue for text-to-speech (TTS) synthesis. |
| Approach: | They propose to use generative language modeling to generate text-to-speech (TTS) outputs by a discrete token-based model. |
| Outcome: | The proposed model is rated higher in naturalness and context appropriateness in listening tests compared to a conventional TTS. |
On Generative Spoken Language Modeling from Raw Audio (2021.tacl-1)
Copied to clipboard
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, Emmanuel Dupoux
| Challenge: | Using a set of metrics to evaluate the learned representations, we aim to create a system that learns from natural interactions as infants learn their first language. |
| Approach: | They propose a task of learning acoustic and linguistic characteristics from raw audio and a set of metrics to evaluate the learned representations at acustic, linguistic and encoding levels. |
| Outcome: | The proposed models evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. |
Efficient Training for Cross-lingual Speech Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Currently, large language models (LLMs) focus on the text modality, making speech modeling difficult. |
| Approach: | They propose a cross-lingual speech language model that trains on discrete speech tokens to achieve cross-modal and cross-linguistic alignment through continual pre-training. |
| Outcome: | The proposed method achieves cross-modal and cross-lingual alignment through continual pre-training. |
Text-Free Prosody-Aware Generative Spoken Language Modeling (2022.acl-long)
Copied to clipboard
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, Wei-Ning Hsu
| Challenge: | Experimental results show that generative spoken language models (LMs) are natural unsupervised multitask learners. |
| Approach: | They propose a prosody-aware generative spoken language model that uses discovered units to generate natural, meaningful, and coherent speech. |
| Outcome: | The proposed model can generate natural, meaningful, and coherent speech given a spoken prompt. |
Continuous Speech Tokenizer in Text To Speech (2025.findings-naacl)
Copied to clipboard
| Challenge: | Autoregressive modeling is a common method for processing language sequences and is effective in token prediction. |
| Approach: | They propose a text-to-speech model based on continuous speech tokens and a continuous tokenizer for speech compression. |
| Outcome: | The proposed model has better continuity and higher estimated Mean Opinion Scores (MoS) this is attributed to better information preservation rate across low and high frequencies in the frequency domain. |
Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer (2024.acl-long)
Copied to clipboard
| Challenge: | Experimental results indicate that GPST significantly outperforms the existing speech language models in terms of word error rate, speech quality, and speaker similarity. |
| Approach: | They propose a hierarchical transformer that quantizes audio waveforms into two distinct types of discrete speech representations and integrates them within a transformer architecture. |
| Outcome: | The proposed model outperforms existing speech language models in word error rate, speech quality, and speaker similarity. |
Language-Codec: Bridging Discrete Codec Representations and Speech Language Models (2025.acl-long)
Copied to clipboard
Shengpeng Ji, Minghui Fang, Jialong Zuo, Ziyue Jiang, Dingdong Wang, Hanting Wang, Hai Huang, Zhou Zhao
| Challenge: | Existing gaps between discrete acoustic codecs and downstream speech language models . initial channel of codebooks contains excessive information, making it difficult to generate tokens from weakly supervised signals such as text. |
| Approach: | They propose a discrete acoustic codec for generating acustic tokens from weakly supervised signals. |
| Outcome: | The proposed language-codec outperforms competing audio compression algorithms and validates on downstream speech language models. |
Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Speech Large Language Models (SpeechLLMs) have emerged as dominant speech processing approaches. |
| Approach: | They compare self-supervised learning-based discrete and continuous features . they compare performance across six spoken language understanding-related tasks . |
| Outcome: | The proposed models outperform discrete tokens and continuous features in six spoken language understanding-related tasks. |
Generative Spoken Dialogue Language Modeling (2023.tacl-1)
Copied to clipboard
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdelrahman Mohamed, Emmanuel Dupoux
| Challenge: | dGSLM is the first “textless” model able to generate audio samples of naturalistic spoken dialogues. |
| Approach: | They propose a model that generates speech, laughter, and other paralinguistic signals in two channels simultaneously and reproduces more naturalistic turn taking compared to a text-based cascaded model. |
| Outcome: | The proposed model reproduces more naturalistic and fluid turn taking than a text-based cascaded model. |
Recent Advances in Speech Language Models: A Survey (2025.acl-long)
Copied to clipboard
Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Steven Y. Guo, Irwin King
| Challenge: | Text-based Large Language Models (LLMs) are a promising solution for end-to-end speech interaction. |
| Approach: | They propose to build a framework that allows users to input text and translate it into speech . they propose to use a text-only LLM and a "textto-speech" framework to generate a response based on this transcription . |
| Outcome: | The survey offers an overview of recent approaches to building SpeechLMs . it outlines core architectural components, training methodologies, evaluation strategies and challenges . |