SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization (2026.acl-long)
Copied to clipboard
Wenxi Chen, Ruiqi Yan, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiquan Li, Yuzhe Liang, null Wenhanlin, Shunshun Yin, Ming Tao, Xinsheng Wang, Xie Chen
| Challenge: | Existing speech codecs struggle to balance high-quality reconstruction with semantically rich representations, limiting their effectiveness in both generative and understanding tasks. |
| Approach: | They propose a neural speech codec with semantic-acoustic dual-stream quantization that disentangles semantic and acousian modeling into two dedicated streams. |
| Outcome: | The proposed codec outperforms state-of-the-art speech tokenizers in auto-propagating text-to-speech models. |
Similar Papers
LLM-Codec: Neural Audio Codec Meets Language Model Objectives (2026.findings-acl)
Copied to clipboard
| Challenge: | Neural audio codecs are optimized for waveform reconstruction rather than autoregressive prediction. |
| Approach: | They propose to augment codec training with language-model-facing objectives while keeping both codec and LLM architectures unchanged. |
| Outcome: | The proposed model improves speech coherence and predictability by preserving the semantic alignment between audio and text representations. |
RepCodec: A Speech Representation Codec for Speech Tokenization (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to discrete speech tokenization, but this discretization can be costly and impedes performance. |
| Approach: | They propose a new speech representation codec for semantic speech tokenization that reconstructs speech representations from speech encoders like HuBERT or data2vec. |
| Outcome: | The proposed method outperforms the widely used k-means clustering approach in speech understanding and generation. |
Language-Codec: Bridging Discrete Codec Representations and Speech Language Models (2025.acl-long)
Copied to clipboard
Shengpeng Ji, Minghui Fang, Jialong Zuo, Ziyue Jiang, Dingdong Wang, Hanting Wang, Hai Huang, Zhou Zhao
| Challenge: | Existing gaps between discrete acoustic codecs and downstream speech language models . initial channel of codebooks contains excessive information, making it difficult to generate tokens from weakly supervised signals such as text. |
| Approach: | They propose a discrete acoustic codec for generating acustic tokens from weakly supervised signals. |
| Outcome: | The proposed language-codec outperforms competing audio compression algorithms and validates on downstream speech language models. |
DM-Codec: Distilling Multimodal Representations for Speech Tokenization (2025.findings-emnlp)
Copied to clipboard
Md Mubtasim Ahasan, Md Fahim, Tasnim Mohiuddin, Akmmahbubur Rahman, Aman Chadha, Tariq Iqbal, M Ashraful Amin, Md Mofijul Islam, Amin Ahsan Ali
| Challenge: | Existing speech tokenization models lack contextual representations for speech synthesis . absence of contextual representation results in elevated WER and WIL scores . |
| Approach: | They propose a language model-guided distillation method that incorporates contextual information into a comprehensive speech tokenizer. |
| Outcome: | The proposed method outperforms state-of-the-art tokenization models in reducing WER and WIL scores. |
AffectCodec: Emotion-Preserving Neural Speech Codec for Expressive Speech Modeling (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing codecs optimize acoustic reconstruction, leaving emotion expressiveness insufficiently modeled at the representation level. |
| Approach: | They propose an emotion-guided neural speech codec that preserves emotional information while maintaining semantic fidelity and prosodic naturalness. |
| Outcome: | The proposed codec preserves emotional cues while maintaining semantic fidelity and prosodic naturalness. |
XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs (2026.acl-long)
Copied to clipboard
Yitian Gong, Luozhijie Jin, Kuangwei Chen, Dong Zhang, Ruifan Deng, Xiaogui Yang, Xin Zhang, Zhaoye Fei, Qinyuan Cheng, Shimin Li, Xipeng Qiu
| Challenge: | Existing speech codecs struggle to balance these objectives at low bitrates . XY-Tokenizer achieves stronger semantic alignment than representative semantic-distillation codec . |
| Approach: | They propose a low-bitrate speech codec that aligns discrete speech representations with text while preserving fine-grained acoustic details for reconstruction. |
| Outcome: | The proposed codec outperforms existing low-bitrate speech codecs in speech understanding and generation tasks. |
ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing neural speech codecs trade model complexity for reconstruction performance . ESC is a lightweight, parameter-efficient speech coder . |
| Approach: | They propose an efficient speech codec based on a cross-scale residual vector quantization scheme and transformers that can achieve high-fidelity speech reconstruction with significantly lower model complexity. |
| Outcome: | The proposed codec achieves high-fidelity speech reconstruction with significantly lower model complexity. |
Analyzing and Mitigating Inconsistency in Discrete Speech Tokens for Neural Codec Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated significant strides in generating high-quality speech . discretizing speech by neural audio codecs often results in sequences that differ from text sequences . |
| Approach: | They quantitatively analyze the Discrete Representation Inconsistency phenomenon within popular audio tokenizers such as EnCodec. |
| Outcome: | The proposed method mitigates the DRI phenomenon within popular audio tokenizers such as EnCodec. |
Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer (2024.acl-long)
Copied to clipboard
| Challenge: | Experimental results indicate that GPST significantly outperforms the existing speech language models in terms of word error rate, speech quality, and speaker similarity. |
| Approach: | They propose a hierarchical transformer that quantizes audio waveforms into two distinct types of discrete speech representations and integrates them within a transformer architecture. |
| Outcome: | The proposed model outperforms existing speech language models in word error rate, speech quality, and speaker similarity. |
Towards Codec-LM Co-design for Neural Codec Language Models (2025.naacl-srw)
Copied to clipboard
| Challenge: | Neural codec language models (or codec LMs) are emerging as a powerful framework for text-to-speech (TTS) despite the close interdependence of codecs and LM, research on codec and lms has largely remained siloed. |
| Approach: | They propose a frame-wise codec encoder that improves both LM log-likelihood and TTS metrics . they also propose LM codebook level dropout to efficiently navigate a portion of codec-LM design space . |
| Outcome: | The proposed codec-LM co-design improves intelligibility, audio quality and speaker control compared to a siloed baseline. |