Improved Speech Representations with Multi-Target Autoregressive Predictive Coding (2020.acl-main)
Copied to clipboard
| Challenge: | Autoregressive coding targets are used to learn meaningful representations from unlabeled speech. |
| Approach: | They propose a method that trains an autoregressive RNN to generate an unseen future frame given a context such as recent past frames. |
| Outcome: | The proposed method can learn representations from unlabeled speech. |
Similar Papers
Mask-Predict: Parallel Decoding of Conditional Masked Language Models (D19-1)
Copied to clipboard
| Challenge: | a masked language model is used to train a model to predict subsets of mangled words . a parallel decoding algorithm can be used to generate translations in a constant number of iterations. |
| Approach: | They propose a model and a parallel decoding algorithm which train a machine to predict any subset of target words . they introduce conditional masked language models (CMLMs) which are trained with a mangled language model objective . |
| Outcome: | The proposed model improves state-of-the-art performance levels for non-autoregressive and parallel decoding models by over 4 BLEU on average. |
FutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue (2023.acl-long)
Copied to clipboard
| Challenge: | Existing pre-trained language models rely on a contrastive framework and are difficult to use in practice. |
| Approach: | They propose a dialogue pre-training model which distills future knowledge to the representation of the previous dialogue context using a self-training framework. |
| Outcome: | The proposed model can be applied to various downstream dialogue tasks. |
Self-supervised Representation Learning for Speech Processing (2022.naacl-tutorials)
Copied to clipboard
Hung-yi Lee, Abdelrahman Mohamed, Shinji Watanabe, Tara Sainath, Karen Livescu, Shang-Wen Li, Shu-wen Yang, Katrin Kirchhoff
| Challenge: | Self-supervised representation learning (SSL) uses proxy supervised learning tasks to obtain training data from unlabeled corpora. |
| Approach: | They propose to survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing to scale up current machine learning technologies. |
| Outcome: | The proposed tutorial is highly relevant to the special theme of ACL about language diversity. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |
Randomized Deep Structured Prediction for Discourse-Level Processing (2021.eacl-main)
Copied to clipboard
| Challenge: | Expressive text encoders have been at the center of recent NLP work . however, some tasks require complex structural dependencies between texts . |
| Approach: | They propose to leverage deep structured prediction and expressive neural encoders for argumentation mining tasks. |
| Outcome: | The proposed framework can be used for argumentation mining tasks without expensive inference tools. |
Augmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models do not produce suitable representations at the discourse level. |
| Approach: | They propose to augment BERT-style language models with a mechanism that allows them to learn suitable discourse-level representations by incorporating top-down connections that operate at the intermediate layers of the network. |
| Outcome: | The proposed approach improves in 6 out of 11 tasks by detecting discourse relationship detection. |
Simple and Effective Unsupervised Speech Translation (2023.acl-long)
Copied to clipboard
Changhan Wang, Hirofumi Inaguma, Peng-Jen Chen, Ilia Kulikov, Yun Tang, Wei-Ning Hsu, Michael Auli, Juan Pino
| Challenge: | Existing methods to train speech models without labeled data are limited for most languages. |
| Approach: | They propose a pipeline approach to build speech translation systems without labeled data by leveraging recent advances in unsupervised speech recognition, machine translation and speech synthesis. |
| Outcome: | The proposed approach outperforms the state-of-the-art in unsupervised speech recognition by 3.2 BLEU on the Libri-Trans benchmark and the best supervised end-to-end models from only two years ago by an average of 5.0 BLUE over five X-En directions. |
Pretraining Methods for Dialog Context Representation Learning (P19-1)
Copied to clipboard
| Challenge: | Existing methods for pretraining dialog context encoders are still in their infancy. |
| Approach: | They propose to use unsupervised pretraining objectives for dialog context representations to fine-tune and evaluate them on a set of downstream dialog tasks. |
| Outcome: | The proposed methods improve performance on a set of dialog tasks and are less data hungry. |
LLM-Codec: Neural Audio Codec Meets Language Model Objectives (2026.findings-acl)
Copied to clipboard
| Challenge: | Neural audio codecs are optimized for waveform reconstruction rather than autoregressive prediction. |
| Approach: | They propose to augment codec training with language-model-facing objectives while keeping both codec and LLM architectures unchanged. |
| Outcome: | The proposed model improves speech coherence and predictability by preserving the semantic alignment between audio and text representations. |
Helping the Weak Makes You Strong: Simple Multi-Task Learning Improves Non-Autoregressive Translators (2022.emnlp-main)
Copied to clipboard
| Challenge: | Non-autoregressive (NAR) neural machine translation models require a conditional independence assumption on target sequences, resulting in less informative learning signals. |
| Approach: | They propose a model-agnostic multi-task learning framework to provide more informative learning signals for NAR models under conventional MLE training. |
| Outcome: | The proposed framework improves accuracy of multiple NAR baselines without additional decoding overhead. |