DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibility (2025.findings-naacl)
Copied to clipboard
| Challenge: | Video-to-speech (V2S) synthesis requires acoustic hints to accurately reconstruct both speech content and speaker characteristics from video clips alone. |
| Approach: | They propose a video-to-speech (V2S) model that predicts Mel-spectrograms directly from video frames. |
| Outcome: | The proposed model outperforms existing models in acoustic intelligibility and preserves speaker-specific characteristics. |
Similar Papers
AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation (2023.acl-long)
Copied to clipboard
Rongjie Huang, Huadai Liu, Xize Cheng, Yi Ren, Linjun Li, Zhenhui Ye, Jinzheng He, Lichao Zhang, Jinglin Liu, Xiang Yin, Zhou Zhao
| Challenge: | Existing models for speech-to-speech translation suffer from distinct degradation in noisy environments and fail to translate visual speech. |
| Approach: | They propose a text-based audio-visual speech-to-speech translation model that integrates visual information with audio-only data to improve system robustness. |
| Outcome: | The proposed model outperforms models trained on audio-only corpus in two languages . it also improves with low-resource audio-visual data, compared with baselines . |
Progressive Facial Granularity Aggregation with Bilateral Attribute-based Enhancement for Face-to-Speech Synthesis (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating speech from facial images rely on pre-trained visual encoders and fine-tune them to align with speech embeddings. |
| Approach: | They propose to derive corresponding voices from facial images using face-to-voice synthesis, which derives corresponding voice from facial image. |
| Outcome: | The proposed approach significantly improves face-voice congruence and synthesis stability. |
Llama-VITS: Enhancing TTS Synthesis with Semantic Awareness (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in Natural Language Processing (NLP) have seen Large-scale Language Models excel at producing high-quality text for various purposes. |
| Approach: | They propose a language model that enriches semantic content of text using Llama2 . their method enhances emotive expressiveness on a dataset . |
| Outcome: | The proposed model matches the naturalness of the original VITS and incorporates BERT (BERT-VITS) on the LJSpeech dataset, highlighting its potential to generate emotive speech. |
TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis (2025.findings-acl)
Copied to clipboard
Yu Zhang, Wenxiang Guo, Changhao Pan, Dongyu Yao, Zhiyuan Zhu, Ziyue Jiang, Yuhan Wang, Tao Jin, Zhou Zhao
| Challenge: | Existing zero-shot singing voice synthesis models depend on phoneme and note boundary annotations, limiting their robustness and producing poor transitions between phonemes and notes. |
| Approach: | They propose a multi-task multilingual zero-shot SVS model with style transfer and style control based on various prompts. |
| Outcome: | Experimental results show that TCSinger 2 outperforms baseline models in subjective and objective metrics across multiple related tasks. |
DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for speech generation are not efficient due to low information density of speech data. |
| Approach: | They propose a method to integrate discrete diffusion models into speech generation tasks . they propose to apply diffusion forward process while employing diffusion backward process . |
| Outcome: | The proposed model achieves comparable results to the auto-regressive baselines with significantly fewer decoding steps (50 steps). |
TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation (2024.findings-acl)
Copied to clipboard
Xize Cheng, Rongjie Huang, Linjun Li, Zehan Wang, Tao Jin, Aoxiong Yin, Chen Feiyang, Xinyu Duan, Baoxing Huai, Zhou Zhao
| Challenge: | Existing methods for talking head translation rely on cascading, resulting in delays and cascadic errors. |
| Approach: | They propose a model for talking head translation, TransFace, which can translate audio-visual speech into audio-visual speech in other languages. |
| Outcome: | The proposed model can translate audio-visual speech into audio-visual speech in other languages. |
SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation (2026.acl-long)
Copied to clipboard
Hui Wang, Jinghua Zhao, Yifan Yang, Shujie Liu, Junyang Chen, Yanzhe Zhang, Shiwan Zhao, Jinyu Li, Jiaming Zhou, Haoqin Sun, Yan Lu, Yong Qin
| Challenge: | Existing methods for evaluating the perceptual quality of synthetic speech are limited due to the complexity of perceptual quality factors and the diversity of speech generation tasks. |
| Approach: | They propose a new paradigm for enabling large language models to conduct structured speech quality evaluation using a large-scale dataset. |
| Outcome: | The proposed model performs well across tasks and languages. |
AudioVSR: Enhancing Video Speech Recognition with Audio Data (2024.emnlp-main)
Copied to clipboard
Xiaoda Yang, Xize Cheng, Jiaqi Duan, Hongshun Qiu, Minjie Hong, Minghui Fang, Shengpeng Ji, Jialong Zuo, Zhiqing Hong, Zhimeng Zhang, Tao Jin
| Challenge: | Recent work has shown poor performance with non-Indo-European languages . previous work primarily utilizes video information to build VSR models . |
| Approach: | They propose a generative model for data inflation that integrates synthetic data with authentic visual data to enhance the VSR model. |
| Outcome: | The proposed model improves on the audio-visual alignment problem in audio-video tasks. |
On Generative Spoken Language Modeling from Raw Audio (2021.tacl-1)
Copied to clipboard
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, Emmanuel Dupoux
| Challenge: | Using a set of metrics to evaluate the learned representations, we aim to create a system that learns from natural interactions as infants learn their first language. |
| Approach: | They propose a task of learning acoustic and linguistic characteristics from raw audio and a set of metrics to evaluate the learned representations at acustic, linguistic and encoding levels. |
| Outcome: | The proposed models evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. |
Simple and Effective Unsupervised Speech Translation (2023.acl-long)
Copied to clipboard
Changhan Wang, Hirofumi Inaguma, Peng-Jen Chen, Ilia Kulikov, Yun Tang, Wei-Ning Hsu, Michael Auli, Juan Pino
| Challenge: | Existing methods to train speech models without labeled data are limited for most languages. |
| Approach: | They propose a pipeline approach to build speech translation systems without labeled data by leveraging recent advances in unsupervised speech recognition, machine translation and speech synthesis. |
| Outcome: | The proposed approach outperforms the state-of-the-art in unsupervised speech recognition by 3.2 BLEU on the Libri-Trans benchmark and the best supervised end-to-end models from only two years ago by an average of 5.0 BLUE over five X-En directions. |