AdaBERT-CTC: Leveraging BERT-CTC for Text-Only Domain Adaptation in ASR (2023.emnlp-industry)
Copied to clipboard
| Challenge: | End-to-end (E2E) automatic speech recognition models struggle to recognize out-of-domain words such as proper nouns and domain-specific terms. |
| Approach: | They propose a domain adaptation technique that relies solely on textual data to adapt to out-of-domain words. |
| Outcome: | The proposed method outperforms the base model by up to 14% relative word error rate improvement on several out-of-domain, publicly available datasets. |
Similar Papers
BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to connectionist temporal classification (CTC) are based on pre-trained language models (LMs) |
| Approach: | They propose a formulation of connectionist temporal classification that relaxes the conditional independence assumptions used in conventional CTC and incorporates linguistic knowledge through explicit output dependency. |
| Outcome: | The proposed model improves over conventional approaches across variations in speaking styles and languages while maintaining CTC’s training efficiency. |
DILBERT: Customized Pre-Training for Domain Adaptation with Category Shift, with an Application to Aspect Extraction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for pre-training can be sub-optimal in some cases . for example, aspect extraction tasks require domain and category invariant representations . |
| Approach: | They propose a domain-invariant learning scheme for BERT to fine-tune pre-trained language models on a source domain and then apply it to a different target domain. |
| Outcome: | The proposed scheme improves performance over state-of-the-art models while using fraction of the unlabeled data. |
CTC-based Non-autoregressive Textless Speech-to-Speech Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing direct speech-to-speech translation models require text supervision during training, which is not feasible for numerous unwritten languages. |
| Approach: | They propose a non-autoregressive (NAR) model that generates discrete units from the source speech and employs a unit-based vocoder to synthesize the target. |
| Outcome: | The proposed model achieves translation quality comparable to the autoregressive model while preserving up to 26.81 decoding speedup. |
Semi-supervised Domain Adaptation for Dependency Parsing via Improved Contextualized Word Representations (2020.coling-main)
Copied to clipboard
| Challenge: | Recent advances in deep neural network models have improved parsing performance on in-domain texts . however, the problem is to improve performance on out-of-domain text data when there is only a small-scale out-domain labeled data. |
| Approach: | They propose to use adversarial learning and fine-tuning BERT to improve contextualized word representations on out-of-domain texts. |
| Outcome: | The proposed models achieve consistent improvement and fine-tune BERT processes boost parsing accuracy by a large margin. |
Type Enhanced BERT for Correcting NER Errors (2023.findings-acl)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is the task of identifying spans that belong to particular categories, such as person, location, organization, etc. |
| Approach: | They propose a method that integrates named entity’s type information into BERT by an adapter layer and integrates it into a gazetteer. |
| Outcome: | The proposed method outperforms baselines in multiple corpus. |
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains (2021.findings-acl)
Copied to clipboard
| Challenge: | Large pre-trained models suffer from domain shift and are not optimal for specific domains. |
| Approach: | They propose a general approach to developing small, fast and effective pretrained models for specific domains by adapting off-the-shelf general pretrained model and performing task-agnostic knowledge distillation in target domains. |
| Outcome: | The proposed approach achieves better performance over the BERT BASE model in domain-specific tasks while 3.3 smaller and 5.1 faster than the BRT BASE. |
Non-Parametric Domain Adaptation for End-to-End Speech Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | End-to-end speech translation (E2E-ST) systems have received increasing attention due to its less error propagation, lower latency and fewer parameters. |
| Approach: | They propose a non-parametric method that leverages in-domain text translation corpus to achieve domain adaptation for E2E-ST systems. |
| Outcome: | The proposed method outperforms the existing in-domain fine-tuning strategies on the Europarl-ST benchmark. |
Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification (2020.lrec-1)
Copied to clipboard
| Challenge: | Aspect-Target Sentiment Classification (ATSC) is a subtask of Aspect Based Sentimence Analysis (ABSA) . recent deep transfer-learning methods have been applied successfully to a myriad of NLP tasks. |
| Approach: | They propose to use a self-supervised domain-specific BERT language model to exploit ATSC . they also perform cross-domain evaluation to explore the real-world robustness of their models . |
| Outcome: | The proposed model outperforms baseline models on the SemEval 2014 task 4 restaurants dataset. |
Attentively Embracing Noise for Robust Latent Representation in BERT (2020.coling-main)
Copied to clipboard
| Challenge: | Modern digital personal assistants interact with users through voice . high error rates still prevail in the widespread adoption of speech technology . |
| Approach: | They propose to extract more robust latent representations for noisy ASR text classification using transformer tokens and attentive embracement layer and multi-head attention layer. |
| Outcome: | The proposed model significantly outperforms the baseline model on the Chatbot and Snips corpora for intent classification with ASR error. |
BERTwich: Extending BERT’s Capabilities to Model Dialectal and Noisy Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models like BERT deteriorate in the face of dialect variation or noise. |
| Approach: | They propose to sandwich BERT's encoder stack between additional encoder layers trained to perform masked language modeling on noisy text. |
| Outcome: | The proposed approach promotes zero-shot transfer to dialectal text and reduces embedding space between words and noisy counterparts. |