Haibin Wu, Ho-Lam Chung, Yi-Cheng Lin, Yuan-Kuei Wu, Xuanjun Chen, Yu-Chi Pai, Hsiu-Hsuan Wang, Kai-Wei Chang, Alexander Liu, Hung-yi Lee
| Challenge: | Researchers have developed a sound codec that can be used as tokenizers for preserving audio data and minimizing data transmission latency. |
| Approach: | They propose to use codec-SUPERB to assess codec models across representative sound applications and signal-level metrics rooted in sound domain knowledge. |
| Outcome: | The proposed codec-SUPERB model is evaluated on selected experimental settings. |
Similar Papers
LLM-Codec: Neural Audio Codec Meets Language Model Objectives (2026.findings-acl)
Copied to clipboard
| Challenge: | Neural audio codecs are optimized for waveform reconstruction rather than autoregressive prediction. |
| Approach: | They propose to augment codec training with language-model-facing objectives while keeping both codec and LLM architectures unchanged. |
| Outcome: | The proposed model improves speech coherence and predictability by preserving the semantic alignment between audio and text representations. |
UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models (2026.acl-demo)
Copied to clipboard
Qundong Shi, Jie Zhou, Biyuan Lin, Junbo Cui, Guoyang Zeng, Yixuan Zhou, Ziyang Wang, Xin Liu, Zhen Luo, Yudong Wang, Zhiyuan Liu
| Challenge: | Existing evaluation frameworks for audio foundation models are heavily reliant on English, making it difficult to objectively assess models’ performance on Chinese. |
| Approach: | They propose a unified framework that supports 10 languages, 14 task categories, 24 models, and 36 benchmarks with one-command evaluation and real-time leaderboards. |
| Outcome: | The proposed framework supports 10 languages, 14 task categories, 24 models, and 36 benchmarks with one-command evaluation and real-time leaderboards. |
Language-Codec: Bridging Discrete Codec Representations and Speech Language Models (2025.acl-long)
Copied to clipboard
Shengpeng Ji, Minghui Fang, Jialong Zuo, Ziyue Jiang, Dingdong Wang, Hanting Wang, Hai Huang, Zhou Zhao
| Challenge: | Existing gaps between discrete acoustic codecs and downstream speech language models . initial channel of codebooks contains excessive information, making it difficult to generate tokens from weakly supervised signals such as text. |
| Approach: | They propose a discrete acoustic codec for generating acustic tokens from weakly supervised signals. |
| Outcome: | The proposed language-codec outperforms competing audio compression algorithms and validates on downstream speech language models. |
UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook (2025.acl-long)
Copied to clipboard
Yidi Jiang, Qian Chen, Shengpeng Ji, Yu Xi, Wen Wang, Chong Zhang, Xianghu Yue, ShiLiang Zhang, Haizhou Li
| Challenge: | Existing neural audio codecs are not capable of handling multi-domain audio data . et al., 2023) integrate speech modality with text-based large language models . |
| Approach: | They propose a unified audio codec with a single codebook to support multi-domain audio data . they propose combining a mix-of-experts strategy and a partitioned domain-adaptive codebook method . |
| Outcome: | The proposed codec outperforms existing codecs on acoustic and semantic representation capabilities. |
Towards Codec-LM Co-design for Neural Codec Language Models (2025.naacl-srw)
Copied to clipboard
| Challenge: | Neural codec language models (or codec LMs) are emerging as a powerful framework for text-to-speech (TTS) despite the close interdependence of codecs and LM, research on codec and lms has largely remained siloed. |
| Approach: | They propose a frame-wise codec encoder that improves both LM log-likelihood and TTS metrics . they also propose LM codebook level dropout to efficiently navigate a portion of codec-LM design space . |
| Outcome: | The proposed codec-LM co-design improves intelligibility, audio quality and speaker control compared to a siloed baseline. |
RepCodec: A Speech Representation Codec for Speech Tokenization (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to discrete speech tokenization, but this discretization can be costly and impedes performance. |
| Approach: | They propose a new speech representation codec for semantic speech tokenization that reconstructs speech representations from speech encoders like HuBERT or data2vec. |
| Outcome: | The proposed method outperforms the widely used k-means clustering approach in speech understanding and generation. |
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities (2022.acl-long)
Copied to clipboard
Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang, Zili Huang, Kushal Lakhotia, Shu-wen Yang, Shuyan Dong, Andy Liu, Cheng-I Lai, Jiatong Shi, Xuankai Chang, Phil Hall, Hsuan-Jui Chen, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, Hung-yi Lee
| Challenge: | Existing evaluation methods for transfer learning are limited in speech research . authors show that pre-trained models transfer well across multiple tasks . |
| Approach: | They propose a benchmark to evaluate pre-trained models by increasing task diversity and difficulty over SUPERB. |
| Outcome: | The proposed benchmark increases task diversity and difficulty over SUPERB-SG. |
BSCodec: A Band-Split Neural Codec for High-Quality Universal Audio Reconstruction (2026.findings-eacl)
Copied to clipboard
| Challenge: | Neural audio codecs have enabled high-fidelity reconstruction of speech, music and sound . however, speech-optimized codec systems suffer degradation on music or sound if they ignore spectral differences . |
| Approach: | They propose a neural audio codec that splits the spectral dimension into separate bands and compresses each band independently. |
| Outcome: | Experimental results show that BSCodec achieves better reconstruction quality on music and sound compared to existing codecs. |
DM-Codec: Distilling Multimodal Representations for Speech Tokenization (2025.findings-emnlp)
Copied to clipboard
Md Mubtasim Ahasan, Md Fahim, Tasnim Mohiuddin, Akmmahbubur Rahman, Aman Chadha, Tariq Iqbal, M Ashraful Amin, Md Mofijul Islam, Amin Ahsan Ali
| Challenge: | Existing speech tokenization models lack contextual representations for speech synthesis . absence of contextual representation results in elevated WER and WIL scores . |
| Approach: | They propose a language model-guided distillation method that incorporates contextual information into a comprehensive speech tokenizer. |
| Outcome: | The proposed method outperforms state-of-the-art tokenization models in reducing WER and WIL scores. |
CoDesc: A Large Code–Description Parallel Dataset (2021.findings-acl)
Copied to clipboard
Masum Hasan, Tanveer Muttaqueen, Abdullah Al Ishtiaq, Kazi Sajeed Mehrab, Md. Mahim Anjum Haque, Tahmid Hasan, Wasi Ahmad, Anindya Iqbal, Rifat Shahriyar
| Challenge: | Existing models for natural language and programming languages are lagging behind due to a lack of large datasets and benchmarks. |
| Approach: | They present a large parallel dataset of Java methods and natural language descriptions that is used to train deep neural models. |
| Outcome: | The proposed dataset improves code summarization and code search by 22% and opens up possibilities for pretrained language models for Java. |