XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs (2026.acl-long)
Copied to clipboard
Yitian Gong, Luozhijie Jin, Kuangwei Chen, Dong Zhang, Ruifan Deng, Xiaogui Yang, Xin Zhang, Zhaoye Fei, Qinyuan Cheng, Shimin Li, Xipeng Qiu
| Challenge: | Existing speech codecs struggle to balance these objectives at low bitrates . XY-Tokenizer achieves stronger semantic alignment than representative semantic-distillation codec . |
| Approach: | They propose a low-bitrate speech codec that aligns discrete speech representations with text while preserving fine-grained acoustic details for reconstruction. |
| Outcome: | The proposed codec outperforms existing low-bitrate speech codecs in speech understanding and generation tasks. |
Similar Papers
LLM-Codec: Neural Audio Codec Meets Language Model Objectives (2026.findings-acl)
Copied to clipboard
| Challenge: | Neural audio codecs are optimized for waveform reconstruction rather than autoregressive prediction. |
| Approach: | They propose to augment codec training with language-model-facing objectives while keeping both codec and LLM architectures unchanged. |
| Outcome: | The proposed model improves speech coherence and predictability by preserving the semantic alignment between audio and text representations. |
RepCodec: A Speech Representation Codec for Speech Tokenization (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to discrete speech tokenization, but this discretization can be costly and impedes performance. |
| Approach: | They propose a new speech representation codec for semantic speech tokenization that reconstructs speech representations from speech encoders like HuBERT or data2vec. |
| Outcome: | The proposed method outperforms the widely used k-means clustering approach in speech understanding and generation. |
SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization (2026.acl-long)
Copied to clipboard
Wenxi Chen, Ruiqi Yan, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiquan Li, Yuzhe Liang, null Wenhanlin, Shunshun Yin, Ming Tao, Xinsheng Wang, Xie Chen
| Challenge: | Existing speech codecs struggle to balance high-quality reconstruction with semantically rich representations, limiting their effectiveness in both generative and understanding tasks. |
| Approach: | They propose a neural speech codec with semantic-acoustic dual-stream quantization that disentangles semantic and acousian modeling into two dedicated streams. |
| Outcome: | The proposed codec outperforms state-of-the-art speech tokenizers in auto-propagating text-to-speech models. |
DM-Codec: Distilling Multimodal Representations for Speech Tokenization (2025.findings-emnlp)
Copied to clipboard
Md Mubtasim Ahasan, Md Fahim, Tasnim Mohiuddin, Akmmahbubur Rahman, Aman Chadha, Tariq Iqbal, M Ashraful Amin, Md Mofijul Islam, Amin Ahsan Ali
| Challenge: | Existing speech tokenization models lack contextual representations for speech synthesis . absence of contextual representation results in elevated WER and WIL scores . |
| Approach: | They propose a language model-guided distillation method that incorporates contextual information into a comprehensive speech tokenizer. |
| Outcome: | The proposed method outperforms state-of-the-art tokenization models in reducing WER and WIL scores. |
Language-Codec: Bridging Discrete Codec Representations and Speech Language Models (2025.acl-long)
Copied to clipboard
Shengpeng Ji, Minghui Fang, Jialong Zuo, Ziyue Jiang, Dingdong Wang, Hanting Wang, Hai Huang, Zhou Zhao
| Challenge: | Existing gaps between discrete acoustic codecs and downstream speech language models . initial channel of codebooks contains excessive information, making it difficult to generate tokens from weakly supervised signals such as text. |
| Approach: | They propose a discrete acoustic codec for generating acustic tokens from weakly supervised signals. |
| Outcome: | The proposed language-codec outperforms competing audio compression algorithms and validates on downstream speech language models. |
Analyzing and Mitigating Inconsistency in Discrete Speech Tokens for Neural Codec Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated significant strides in generating high-quality speech . discretizing speech by neural audio codecs often results in sequences that differ from text sequences . |
| Approach: | They quantitatively analyze the Discrete Representation Inconsistency phenomenon within popular audio tokenizers such as EnCodec. |
| Outcome: | The proposed method mitigates the DRI phenomenon within popular audio tokenizers such as EnCodec. |
Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Speech Language Models (LSLMs) typically operate at high token rates to ensure acoustic fidelity, yet this results in sequence lengths that exceed the underlying semantic content, incurring prohibitive inference costs. |
| Approach: | They propose a token-based token merging mechanism that uses a training-free token pooling mechanism to reduce prefilling FLOPs by 27.48% while maintaining competitive accuracy. |
| Outcome: | The proposed method reduces prefilling FLOPs by 27.48% while maintaining competitive accuracy. |
AffectCodec: Emotion-Preserving Neural Speech Codec for Expressive Speech Modeling (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing codecs optimize acoustic reconstruction, leaving emotion expressiveness insufficiently modeled at the representation level. |
| Approach: | They propose an emotion-guided neural speech codec that preserves emotional information while maintaining semantic fidelity and prosodic naturalness. |
| Outcome: | The proposed codec preserves emotional cues while maintaining semantic fidelity and prosodic naturalness. |
TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) for code rely on subword tokenizers learned from mixed natural language text and programming language code but driven by statistics rather than grammar. |
| Approach: | They propose a framework that applies semantic-preserving rewrite rules to create code variants differing only in tokenization. |
| Outcome: | The proposed framework can create code variants differing only in tokenization . the findings highlight the need for grammar-aware tokenization for future code LLMs. |
How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them (2026.acl-long)
Copied to clipboard
| Challenge: | Tokenization is the first step in every language model (LM), yet it never takes the sounds of words into account. |
| Approach: | They propose a lightweight IPA-based fine-tuning method that infuses phonological awareness into LMs. |
| Outcome: | The proposed method improves phonological awareness across three phonology-related tasks while preserving math and general reasoning ability. |