Papers by Sravan Bodapati
Multi-teacher Distillation for Multilingual Spelling Correction (2023.emnlp-industry)
Copied to clipboard
| Challenge: | a multilingual spelling correction model is needed to meet the tight latency requirements of multilingual NLP . a monolingual teacher model is trained for each language/locale, and individual models are distilled into a single student model . |
| Approach: | They propose a multilingual approach to spelling correction using multi-teacher distillation . they train a monolingual teacher model for each language and distill them into a single model . |
| Outcome: | The proposed model can meet the tight latency requirements of deployed services. |
Masked Audio Text Encoders are Effective Multi-Modal Rescorers (2023.findings-acl)
Copied to clipboard
| Challenge: | Masked Language Models (MLMs) have proven to be effective for second-pass rescoring in Automatic Speech Recognition systems. |
| Approach: | They propose a multi-modal masked language model rescorer which integrates acoustic representations into the input space of MLM. |
| Outcome: | The proposed model reduces word error rate (WER) by 4%-16% on in-domain and 3%-7% on out-of-domain datasets over the text-only baseline. |
AdaBERT-CTC: Leveraging BERT-CTC for Text-Only Domain Adaptation in ASR (2023.emnlp-industry)
Copied to clipboard
| Challenge: | End-to-end (E2E) automatic speech recognition models struggle to recognize out-of-domain words such as proper nouns and domain-specific terms. |
| Approach: | They propose a domain adaptation technique that relies solely on textual data to adapt to out-of-domain words. |
| Outcome: | The proposed method outperforms the base model by up to 14% relative word error rate improvement on several out-of-domain, publicly available datasets. |
Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale (2023.acl-long)
Copied to clipboard
| Challenge: | 70% of attention heads and 20% of the feed forward networks can be removed with minimal decline in task performance. |
| Approach: | They propose to investigate whether in-context learning is not uniform across all components of a large language model. |
| Outcome: | The proposed model can remove 70% of attention heads and 20% of feed forward networks with minimal decline in task performance. |
Robustness to Capitalization Errors in Named Entity Recognition (D19-55)
Copied to clipboard
| Challenge: | Existing methods to improve robustness to noise discard given orthographic information, which significantly degrades models' performance on well-formed text. |
| Approach: | They propose a method which allows models to learn to utilize or ignore orthographic information depending on its usefulness in the context. |
| Outcome: | The proposed approach achieves competitive robustness to capitalization errors while making negligible compromises on well-formed text and significantly improving generalization power on noisy user-generated text. |
Retrieve and Copy: Scaling ASR Personalization to Large Catalogs (2023.emnlp-industry)
Copied to clipboard
| Challenge: | End-to-end ASR models struggle to recognize uncommon domain-specific words due to limited audio context. |
| Approach: | They propose a "Retrieve and Copy" mechanism to improve latency while retaining the accuracy even when scaled to a large catalog. |
| Outcome: | The proposed method achieves 6% more word error rate reduction and 3.6% improvement in F1 when scaled to a large catalog size while retaining the accuracy. |