Impact Analysis of the Use of Speech and Language Models Pretrained by Self-Supersivion for Spoken Language Understanding

Challenge:	Pretrained models have been introduced for both acoustic and language modeling.
Approach:	They present an error analysis of pretrained models using a french MEDIA benchmark dataset.
Outcome:	The proposed models have been able to improve on the french MEDIA benchmark dataset, which is one of the most challenging among all benchmarks accessible to the entire research community.

Similar Papers

Overview

Challenge

Approach

Outcome

wav2vec-S: Adapting Pre-trained Speech Models for Streaming (2024.findings-acl)

Copied to clipboard

Biao Fu, Kai Fan, Minpeng Liao, Yidong Chen, Xiaodong Shi, Zhongqiang Huang

Text Generation

Challenge:	Pre-trained speech models have advanced speech-related tasks, including speech recognition and translation.
Approach:	They propose a pre-trained speech model that incorporates modifications to ensure consistent speech representations during training and inference phases for streaming speech inputs.
Outcome:	The proposed model outperforms baseline models on speech recognition and translation tasks and achieves a superior balance between quality and latency.

The Spoken Language Understanding MEDIA Benchmark Dataset in the Era of Deep Learning: data updates, training and evaluation tools (2022.lrec-1)

Copied to clipboard

Gaëlle Laperrière, Valentin Pelloin, Antoine Caubrière, Salima Mdhaffar, Nathalie Camelin, Sahar Ghannay, Bassam Jabaian, Yannick Estève

Multimodality

Challenge:	a growing number of studies address the spoken language understanding domain through a simple task like speech intent detection.
Approach:	They focus on the french MEDIA SLU dataset, which is distributed since 2005 . they propose a recipe for its use, including data preparation, training and evaluation scripts .
Outcome:	The MEDIA SLU dataset is used as a benchmark dataset for a large number of research projects.

SpeechLLMs for Large-scale Contextualized Zero-shot Slot Filling (2025.emnlp-industry)

Copied to clipboard

Kadri Hacioglu, Manjunath K E, Andreas Stolcke

Semantic Text Processing

Challenge:	Slot filling is a key subtask in spoken language understanding (SLU) . recent advent of speech-based large language models has opened new avenues for speech understanding .
Approach:	They propose to improve slot-filling task by creating an empirical upper bound for the task . they propose to use a speech-based large language model to integrate speech and text modalities .
Outcome:	The proposed model improves slot filling performance while reducing generalization gaps.

New Semantic Task for the French Spoken Language Understanding MEDIA Benchmark (2024.lrec-main)

Copied to clipboard

Nadège Alavoine, Gaëlle Laperrière, Christophe Servan, Sahar Ghannay, Sophie Rosset

Information Retrieval

Challenge:	Intent classification and slot-filling tasks are essential tasks of Spoken Language Understanding (SLU).
Approach:	They propose to use a MEDIA SLU dataset to train a multilingual model to achieve both tasks jointly.
Outcome:	The proposed model can be trained on multiple datasets including the MEDIA dataset and extends to more tasks and use cases.

On the Evaluation of Speech Foundation Models for Spoken Language Understanding (2024.findings-acl)

Copied to clipboard

Siddhant Arora, Ankita Pasad, Chung-Ming Chien, Jionghao Han, Roshan Sharma, Jee-weon Jung, Hira Dhamyal, William Chen, Suwon Shon, Hung-yi Lee, Karen Livescu, Shinji Watanabe

Multimodality

Challenge:	Spoken language understanding evaluation (SLUE) benchmarks are used to benchmark complex spoken language understanding tasks on natural speech.
Approach:	They propose a set of benchmark tasks to evaluate spoken language understanding on natural speech . they use pre-trained speech foundation models to evaluate the utility of different SFMs .
Outcome:	The proposed framework outperforms pre-trained speech foundation models on natural speech . the proposed framework also outperformed self-supervised SFMs on the sequence generation tasks .

Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs (2021.tacl-1)

Copied to clipboard

Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, Desmond Elliott

Multimodality

Challenge:	Large-scale pretraining and task-specific fine-tuning are now the standard methodology for many tasks in computer vision and natural language processing.
Approach:	They propose to combine two types of vision and language BERTs to create a theoretical framework that can be unified under different theoretical frameworks.
Outcome:	The proposed models can be classified into single-stream or dual-stream encoders and are unified under a single theoretical framework.

Frustratingly Simple Pretraining Alternatives to Masked Language Modeling (2021.emnlp-main)

Copied to clipboard

Atsuki Yamaguchi, George Chrysostomou, Katerina Margatina, Nikolaos Aletras

Language Models

Challenge:	Masked language modeling (MLM) is widely used in natural language processing for self-supervised learning of text representations.
Approach:	They propose to use token-level classification tasks as main pretraining objectives instead of Masked language modeling (MLM) . Empirical results show that pretraining a model with 41% of the BERT-BASE’s parameters, BERT MEDIUM results in only a 1% drop in GLUE scores with their best objective.
Outcome:	Empirical results show that the proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture.

Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)

Copied to clipboard

Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman

Language Models

Challenge:	State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text.
Approach:	They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling.
Outcome:	The proposed model can be used to train sentences on language modeling tasks.

Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning (2024.findings-acl)

Copied to clipboard

Yang Zhao, Li Du, Xiao Ding, Kai Xiong, Zhouhao Sun, Shi Jun, Ting Liu, Bing Qin

Language Models

Challenge:	Existing studies have suggested that the composition of the pretraining corpus exerts a significant impact upon the performance of LLMs.
Approach:	They analyze the impact of 48 datasets from 5 major categories of pretraining data of Large Language Models and measure their impacts on LLMs using benchmarks about nine major categories.
Outcome:	The proposed analysis provides insights into the organization of data to support more efficient pretraining of Large Language Models.

On the importance of pre-training data volume for compact language models (2020.emnlp-main)

Copied to clipboard

Vincent Micheli, Martin d’Hoffschmidt, François Fleuret

Language Models

Challenge:	Recent advances in language modeling have led to computationally intensive and resource-demanding state-of-the-art models.
Approach:	They investigate the impact of pre-training data volume on compact language models . they use a French question answering task to train models with as little as 100 MB of text .
Outcome:	The results show that pre-training data volume can improve models with as little as 100 MB of text . the results suggest that the model performance is poorer with less data than with larger datasets .