An Effective Automated Speaking Assessment Approach to Mitigating Data Scarcity and Imbalanced Distribution

Challenge:	Automated speaking assessment (ASA) typically involves automatic speech recognition (ASR) and hand-crafted feature extraction from the transcript of a learner’s speech.
Approach:	They propose to use metric-based classification and loss re-weighting to model the impact of different SSL-based embedding features on the CEFR score.
Outcome:	The proposed model outperforms baselines on the ICNALE benchmark dataset, achieving a significant improvement of more than 10% in CEFR prediction accuracy.

Similar Papers

Overview

Challenge

Approach

Outcome

Self-supervised Representation Learning for Speech Processing (2022.naacl-tutorials)

Copied to clipboard

Hung-yi Lee, Abdelrahman Mohamed, Shinji Watanabe, Tara Sainath, Karen Livescu, Shang-Wen Li, Shu-wen Yang, Katrin Kirchhoff

Semantic Text Processing

Challenge:	Self-supervised representation learning (SSL) uses proxy supervised learning tasks to obtain training data from unlabeled corpora.
Approach:	They propose to survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing to scale up current machine learning technologies.
Outcome:	The proposed tutorial is highly relevant to the special theme of ACL about language diversity.

WER-BERT: Automatic WER Estimation with BERT in a Balanced Ordinal Classification Paradigm (2021.eacl-main)

Copied to clipboard

Akshay Krishna Sheshadri, Anvesh Rao Vijjini, Sukhdeep Kharbanda

Information Extraction & Text Mining

Challenge:	Automatic Speech Recognition (ASR) systems are evaluated using Word Error Rate (WER) a higher WER means a lower percentage of errors between the ground truth and the transcription of the system.
Approach:	They propose a new balanced paradigm for automatic Word Error Rate estimation using a Librispeech dataset and a Google Cloud's Speech-to-Text API.
Outcome:	The proposed approach is more effective than regression in a classification setting, but suffers from heavy class imbalance.

MCLF: A Multi-grained Contrastive Learning Framework for ASR-robust Spoken Language Understanding (2023.findings-emnlp)

Copied to clipboard

Zhiqi Huang, Dongsheng Chen, Zhihong Zhu, Xuxin Cheng

Semantic Text Processing

Challenge:	Trending ASR-robust SLU systems have seen impressive improvements through global contrastive learning, but they can easily lead to severe semantic changes.
Approach:	They propose a two-stage multi-grained contrastive learning framework to improve ASR robustness . they first adapt pre-trained language models to downstream SLU datasets and then fine-tune it on the corresponding dataset.
Outcome:	The proposed framework improves on four datasets and four BERT-like backbone models.

Error-preserving Automatic Speech Recognition of Young English Learners’ Language (2024.acl-long)

Copied to clipboard

Janick Michot, Manuela Hürlimann, Jan Deriu, Luzia Sauer, Katsiaryna Mlynchyk, Mark Cieliebak

Multimodality

Challenge:	State-of-the-art speech recognition models are often trained on adult read-aloud data by native speakers and do not transfer well to young language learners’ speech.
Approach:	They propose to use an automated speech recognition module to train language learners' speaking skills on spontaneous speech by young language learners.
Outcome:	The proposed model improves on 85 hours of English audio spoken by Swiss learners and preserves their mistakes.

Exploring Pathological Speech Quality Assessment with ASR-Powered Wav2Vec2 in Data-Scarce Context (2024.lrec-main)

Copied to clipboard

Tuan Nguyen, Corinne Fredouille, Alain Ghio, Mathieu Balaguer, Virginie Woisard

Multimodality

Challenge:	Current studies only gain good results on simple tasks such as binary classification due to data scarcity.
Approach:	They propose to use the pre-trained Wav2Vec2 architecture for both SSL, and ASR as feature extractor in speech assessment.
Outcome:	The proposed system achieves the best results on the HNC dataset using 95 training samples.

Discourse on ASR Measurement: Introducing the ARPOCA Assessment Tool (2022.acl-srw)

Copied to clipboard

Megan Merz, Olga Scrivner

Speech & Audio in NLP

Challenge:	Automated speech recognition (ASR) models are based on a corpus of audio recordings, but are often small or nonexistent for less common languages and dialects.
Approach:	This research proposal will develop a semi-automatic acoustic features extraction system that integrates phonetic transcripts with pronunciation dictionaries.
Outcome:	The proposed system will be used to improve language recognition and model feedback in less common languages and dialects.

MoE-SLU: Towards ASR-Robust Spoken Language Understanding via Mixture-of-Experts (2024.findings-acl)

Copied to clipboard

Xuxin Cheng, Zhihong Zhu, Xianwei Zhuang, Zhanpeng Chen, Zhiqi Huang, Yuexian Zou

Robustness in NLP

Challenge:	Spoken language understanding (SLU) is a crucial task in task-oriented dialogue systems.
Approach:	They propose an ASR-Robust SLU framework based on the mixture-of-experts technique to generate additional transcripts from clean transcripts and use it to weigh the representations of the generated transcripts, ASR transcripts .
Outcome:	The proposed framework achieves state-of-the-art on three benchmark SLU datasets.

Evaluating Self-Supervised Speech Representations for Indigenous American Languages (2024.lrec-main)

Copied to clipboard

Chih-Chen Chen, William Chen, Rodolfo Joel Zevallos, John E. Ortega

Multimodality

Challenge:	a recent study focused on the use of self-supervised learning to learn speech representations for indigenous languages . aaron e. scott: the vast linguistic diversity represented by indigenous languages remains unexplored . by expanding the scope of language processing to include indigenous languages, we can foster linguistic inclusivity, he says .
Approach:	They benchmark the efficacy of large-scale self-supervised learning models on indigenous American languages.
Outcome:	The proposed model can generalize to real-world data, showing strong performance . evaluators found that the model performed better than monolingual models on indigenous languages .

Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech (2024.naacl-long)

Copied to clipboard

Dena Mujtaba, Nihar Mahapatra, Megan Arney, J Yaruss, Hope Gerlach-Houck, Caryn Herring, Jia Bin

Multimodality

Challenge:	Automatic speech recognition systems fail to accurately interpret speech patterns deviating from typical fluency, leading to critical usability issues and misinterpretations.
Approach:	They evaluate six leading automatic speech recognition systems based on a real-world dataset and a synthetic dataset derived from the widely-used LibriSpeech benchmark.
Outcome:	The six leading speech recognition systems were evaluated on a real-world dataset and a synthetic dataset derived from the widely-used LibriSpeech benchmark.

CEASR: A Corpus for Evaluating Automatic Speech Recognition (2020.lrec-1)

Copied to clipboard

Malgorzata Anna Ulasik, Manuela Hürlimann, Fabian Germann, Esin Gedik, Fernando Benites, Mark Cieliebak

Multimodality

Challenge:	Automatic Speech Recognition (ASR) systems are increasingly needed for research and practical applications.
Approach:	They propose to use public speech corpora to evaluate the quality of automatic speech recognition (ASR) they calculate an average Word Error Rate (WER) per corpus, per system and per corpor-system pair .
Outcome:	The proposed corpus evaluates the quality of automatic speech recognition systems using public speech corpora and transcripts generated by state-of-the-art systems.