Learning Paralinguistic Features from Audiobooks through Style Voice Conversion (2021.naacl-main)
Copied to clipboard
| Challenge: | Paralinguistics, the non-lexical components of speech, play a crucial role in human-human interaction. |
| Approach: | They propose a framework that enables a neural network to learn to extract paralinguistic attributes from speech using data that are not annotated for emotion. |
| Outcome: | The proposed framework improves on emotion recognition and speaking style detection tasks. |
Similar Papers
On the Emotion Understanding of Synthesized Speech (2026.acl-long)
Copied to clipboard
Yuan Ge, Haishu Zhao, AoKai Hao, Junxiang Zhang, Bei Li, Xiaoqian Liu, Chenglong Wang, Jianjin Wang, Bingsen Zhou, Bingyu Liu, JingBo Zhu, Zhengtao Yu, Tong Xiao
| Challenge: | Existing models for emotion understanding do not capture fundamental features of synthesized speech. |
| Approach: | They evaluate emotion recognition models on synthesized speech using SER models and generative models. |
| Outcome: | The proposed model can't generalize to synthesized speech because of speech token prediction . generative models tend to infer emotion from textual semantics while ignoring paralinguistic cues. |
Audio-Based Linguistic Feature Extraction for Enhancing Multi-lingual and Low-Resource Text-to-Speech (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to synthesize speech for low-resource languages require a substantial amount of source language corpora to generate the linguistic knowledge that can be reused for speech synthesis. |
| Approach: | They propose a method that extracts linguistic features from audio input while effectively filtering out miscellaneous acoustic information including speaker-specific attributes like timbre. |
| Outcome: | The proposed method extracts linguistic features from audio input while effectively filtering out miscellaneous acoustic information including speaker-specific attributes like timbre. |
Explaining Speech Classification Models via Word-Level Audio Segments and Paralinguistic Features (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing explanations for speech classification models are difficult to interpret and make mistakes. |
| Approach: | They propose to explain speech classification models by using word-level and paralinguistic attributes to measure the impact of each audio segment aligned with a word on the outcome. |
| Outcome: | The proposed explanations correctly represent the model’s inner workings and are plausible to humans. |
Parsing Speech: a Neural Approach to Integrating Lexical and Acoustic-Prosodic Information (N18-1)
Copied to clipboard
| Challenge: | acoustic signals provide cues that help listeners disambiguate difficult parses . speech carries useful extra information associated with prosodic structure . |
| Approach: | They propose a model that integrates transcribed text and acoustic-prosodic features into a neural network that accepts text and prosodic feature. |
| Outcome: | The proposed model improves parse and disfluency detection scores over a strong text-only baseline. |
Accented Speech Recognition With Accent-specific Codebooks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Degradation in performance across underrepresented accents is a severe deterrent to inclusive adoption of ASR. |
| Approach: | They propose an approach to adapt speech accents to unseen accents by using cross-attention with a trainable set of codebooks. |
| Outcome: | The proposed approach yields significant performance gains on the seen English accents and unseen accents on the Mozilla Common Voice dataset. |
Textless Speech Emotion Conversion using Discrete & Decomposed Representations (2022.emnlp-main)
Copied to clipboard
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu Anh Nguyen, Morgan Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, Yossi Adi
| Challenge: | Existing methods for modifying emotion of speech are difficult because emotion affects all levels simultaneously. |
| Approach: | They propose a method to convert a spoken language speech into a model of emotion . they use phonetic-content units, prosodic features, speaker, and emotion to modify the emotion a speech utterance has. |
| Outcome: | The proposed method beats text-based systems in terms of perceived emotion and audio quality. |
Scaling Rich Style-Prompted Text-to-Speech Datasets (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets that only cover basic tags are limited in their scale or coverage of style tags. |
| Approach: | They propose a large-scale dataset that annotates speech utterances with rich style captions. |
| Outcome: | The proposed dataset scales speech utterances with rich style captions for the first time. |
The importance of fillers for text representations of speech transcripts (2020.emnlp-main)
Copied to clipboard
| Challenge: | Fillers are a type of disfluency that can be a sound ("um" or "uh") filling a pause in an utterance or conversation. |
| Approach: | They propose to represent fillers with deep contextualised embeddings to improve modelling of spoken language and two downstream tasks . |
| Outcome: | The proposed representations improve modelling of spoken language and two downstream tasks, predicting a speaker’s stance and expressed confidence. |
Automatic Dialogue Generation with Expressed Emotions (N18-2)
Copied to clipboard
| Challenge: | a growing interest in neural dialogue generation systems is focusing on generating human-like responses based on past utterances . despite efforts, few consider putting restrictions on the response itself . authors present three models that concatenate the desired emotion with the source input . |
| Approach: | They propose three models that concatenate the desired emotion with the source input or push the emotion in the decoder. |
| Outcome: | The proposed model is more efficient than the previous models, but it lacks the emotion vector. |
Self-supervised Cross-modal Pretraining for Speech Emotion Recognition and Sentiment Analysis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multimodal speech emotion recognition and sentiment analysis have not improved results due to their relatively simple fusion mechanisms and lack of proper cross-modal pretraining. |
| Approach: | They propose a deep-fused audio-text bi-modal transformer with carefully designed cross-modal fusion mechanism and stage-wise cross-mod pretraining scheme to facilitate cross-modulation. |
| Outcome: | The proposed method exceeds benchmarks on public IEMOCAP emotion and CMU-MOSEI sentiment datasets by a large margin. |