Improving Acoustic Word Embeddings through Correspondence Training of Self-supervised Speech Representations (2024.eacl-long)
Copied to clipboard
| Challenge: | Acoustic word embeddings are vector representations of spoken words . self-supervised learning (SSL)-based speech models are popular for speech recognition . |
| Approach: | They explore the effectiveness of the Correspondence Auto-Encoder to obtain improved AWEs . they use self-supervised learning on large amounts of unlabelled speech data . |
| Outcome: | The Correspondence Auto-Encoder (CAE) model outperforms MFCC models on language discrimination . the model achieves best results in Polish, Portuguese, Spanish, French, and English . |
Similar Papers
Self-supervised Representation Learning for Speech Processing (2022.naacl-tutorials)
Copied to clipboard
Hung-yi Lee, Abdelrahman Mohamed, Shinji Watanabe, Tara Sainath, Karen Livescu, Shang-Wen Li, Shu-wen Yang, Katrin Kirchhoff
| Challenge: | Self-supervised representation learning (SSL) uses proxy supervised learning tasks to obtain training data from unlabeled corpora. |
| Approach: | They propose to survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing to scale up current machine learning technologies. |
| Outcome: | The proposed tutorial is highly relevant to the special theme of ACL about language diversity. |
On the Language Encoder of Contrastive Cross-modal Models (2024.findings-acl)
Copied to clipboard
Mengjie Zhao, Junya Ono, Zhi Zhong, Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Wei-Hsiang Liao, Takashi Shibuya, Hiromi Wakaki, Yuki Mitsufuji
| Challenge: | Pretrained audio-language models such as AudioCLIP and AudioCLAP have shown promising results on vision-language (VL) tasks. |
| Approach: | They extensively evaluate how unsupervised and supervised sentence embedding training affect language encoder quality and cross-modal task performance. |
| Outcome: | The proposed model improves on visual-language (VL) and audio-language tasks when the amount of training data is large. |
Language-Codec: Bridging Discrete Codec Representations and Speech Language Models (2025.acl-long)
Copied to clipboard
Shengpeng Ji, Minghui Fang, Jialong Zuo, Ziyue Jiang, Dingdong Wang, Hanting Wang, Hai Huang, Zhou Zhao
| Challenge: | Existing gaps between discrete acoustic codecs and downstream speech language models . initial channel of codebooks contains excessive information, making it difficult to generate tokens from weakly supervised signals such as text. |
| Approach: | They propose a discrete acoustic codec for generating acustic tokens from weakly supervised signals. |
| Outcome: | The proposed language-codec outperforms competing audio compression algorithms and validates on downstream speech language models. |
Emergent morpho-phonological representations in self-supervised speech models (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that self-supervised speech models do not represent phonological and morphological phenomena in frequent English noun and verb inflections. |
| Approach: | They study how S3Ms represent phonological and morphological phenomena in English . they propose alternative representational strategies that may support human spoken word recognition . |
| Outcome: | a new study shows that S3M models can represent phonological and morphological phenomena in English . the models can be trained to recognize spoken words in naturalistic, noisy environments . |
Causal2Vec: Improving Decoder-only LLMs as Embedding Models through a Contextual Token (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods modify attention mechanism to be bidirectional, undermining LLMs’ ability to extract semantic information acquired during pre-training. |
| Approach: | They propose a general-purpose embedding model that pre-encodes input text into a single Contextual token and then prepends it to the LLM's input sequence. |
| Outcome: | The proposed model improves performance of decoder-only large language models without altering their architectures or introducing significant computational overhead. |
Dissecting Contextual Word Embeddings: Architecture and Representation (D18-1)
Copied to clipboard
| Challenge: | Existing work on learning contextual representations has used LSTM-based biLMs, but there is no reason to believe this is effective. |
| Approach: | They propose to use pre-trained bidirectional language models to learn contextual word embeddings for four NLP tasks and to use them to study the effects of architecture on endtask accuracy. |
| Outcome: | The proposed models outperform word embeddings for four NLP tasks and all learn representations that vary with network depth. |
MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery (2026.acl-long)
Copied to clipboard
| Challenge: | MauBERT models learn from multilingual data to predict articulatory features or phones, resulting in language-independent phonetic representations. |
| Approach: | They introduce a multilingual extension of HuBERT that leverages articulatory features for robust cross-lingual phonetic representation learning. |
| Outcome: | The proposed model can predict phonetic features in 55 languages with minimal fine-tuning (10 hours of speech) it is more context-invariant than state-of-the-art models and adapts to unseen languages and casual speech with minimal self-supervised fine- tuning (10 hours) |
Evaluating Self-Supervised Speech Representations for Indigenous American Languages (2024.lrec-main)
Copied to clipboard
| Challenge: | a recent study focused on the use of self-supervised learning to learn speech representations for indigenous languages . aaron e. scott: the vast linguistic diversity represented by indigenous languages remains unexplored . by expanding the scope of language processing to include indigenous languages, we can foster linguistic inclusivity, he says . |
| Approach: | They benchmark the efficacy of large-scale self-supervised learning models on indigenous American languages. |
| Outcome: | The proposed model can generalize to real-world data, showing strong performance . evaluators found that the model performed better than monolingual models on indigenous languages . |
LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained speech encoders and large language models can perform suboptimal performance on a range of spoken language processing tasks. |
| Approach: | They propose to combine large-scale pre-trained speech encoders and large-language models for better performance on automatic speech recognition tasks. |
| Outcome: | The proposed model can get an average of 49% WER reduction over the baseline model on 8 MLS testsets. |
RepCodec: A Speech Representation Codec for Speech Tokenization (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to discrete speech tokenization, but this discretization can be costly and impedes performance. |
| Approach: | They propose a new speech representation codec for semantic speech tokenization that reconstructs speech representations from speech encoders like HuBERT or data2vec. |
| Outcome: | The proposed method outperforms the widely used k-means clustering approach in speech understanding and generation. |