Papers with E2E
Spoken Conversational Agents with Large Language Models (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial focuses on the evolution of voice-native LLMs . it reviews the adaptation of text LLM to audio, cross-modal alignment, and joint speech–text training . |
| Approach: | This tutorial examines the evolution of voice-native LLMs in conversational agents . it compares cascaded and voice-based LLM systems to end-to-end retrieval-and vision-grounded systems . |
| Outcome: | This tutorial examines the evolution of voice-native LLMs . it compares the performance of voice assistants to current open-domain agents . |
Exploiting BERT for End-to-End Aspect-based Sentiment Analysis (D19-55)
Copied to clipboard
| Challenge: | Existing studies on ABSA use a sequence tagging problem to extract aspect-specific opinion words from the sentence given the aspect. |
| Approach: | They build a series of simple yet insightful neural baselines to deal with E2E-ABSA task using contextualized embeddings from pre-trained language models. |
| Outcome: | The proposed architecture outperforms state-of-the-art models even with a simple linear classification layer. |
An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for speech-to-text translation (ST) have achieved impressive supervised and zero-shot performance. |
| Approach: | They propose to use consistency regularization methods to boost end-to-end (E2E) speech-totext translation (ST) by regularizing the intra-modal consistency instead of the modality gap. |
| Outcome: | The proposed training strategies achieve state-of-the-art (SOTA) performance in most translation directions. |
Non-Parametric Domain Adaptation for End-to-End Speech Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | End-to-end speech translation (E2E-ST) systems have received increasing attention due to its less error propagation, lower latency and fewer parameters. |
| Approach: | They propose a non-parametric method that leverages in-domain text translation corpus to achieve domain adaptation for E2E-ST systems. |
| Outcome: | The proposed method outperforms the existing in-domain fine-tuning strategies on the Europarl-ST benchmark. |
AdaBERT-CTC: Leveraging BERT-CTC for Text-Only Domain Adaptation in ASR (2023.emnlp-industry)
Copied to clipboard
| Challenge: | End-to-end (E2E) automatic speech recognition models struggle to recognize out-of-domain words such as proper nouns and domain-specific terms. |
| Approach: | They propose a domain adaptation technique that relies solely on textual data to adapt to out-of-domain words. |
| Outcome: | The proposed method outperforms the base model by up to 14% relative word error rate improvement on several out-of-domain, publicly available datasets. |
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning (2021.acl-long)
Copied to clipboard
| Challenge: | Existing vision-language pre-training methods use a two-step training procedure to learn visual features from image-text pairs. |
| Approach: | They propose a vision-language pre-trained model for V+L understanding and generation using a unified Transformer framework. |
| Outcome: | The proposed model can learn visual representation and semantic alignments between image and text on visual-text pairs and on visual processing tasks. |
On the Use of External Data for Spoken Named Entity Recognition (2022.naacl-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) tasks require large labeled datasets to perform . compared to prior work, relative improvements in F1 of up to 16% are found . |
| Approach: | They propose to use self-training, knowledge distillation, and transfer learning to learn SLU models . they compare pipeline and pipeline approaches to find out how to use external data . |
| Outcome: | The proposed models improve performance beyond pre-trained models in resource-constrained settings . the best baseline model is a pipeline approach, while the best performance is achieved by an E2E model. |
Building Adaptive Acceptability Classifiers for Neural NLG (2021.emnlp-main)
Copied to clipboard
Soumya Batra, Shashank Jain, Peyman Heidari, Ankit Arun, Catharine Youngs, Xintong Li, Pinar Donmez, Shawn Mei, Shiunzu Kuo, Vikas Bhardwaj, Anuj Kumar, Michael White
| Challenge: | Existing approaches to generate synthetic data using simple sentence transformations and/or model-based techniques may not generate realistic error samples with respect to the NLG models. |
| Approach: | They propose a framework to train models to classify acceptability of responses generated by natural language generation models using a 2-stage approach . they use existing sentence transformations to generate samples that better resemble the output of the generation models. |
| Outcome: | The proposed approach outperforms existing techniques and can be used in few-shot settings using self-training. |
End-to-End Speech to Intent Prediction to improve E-commerce Customer Support Voicebot in Hindi and English (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Experimental results show that an E2E S2I model outperforms a conventional pipeline by 27% on the F1 score . multi-component pipelines require large annotated datasets, have higher latency, and have complex deployment. |
| Approach: | They propose an end-to-end (E2E) S2I model that uses pre-trained automatic speech recognition with slight modification and fine-tuning on small annotated datasets. |
| Outcome: | The proposed model outperforms a conventional pipeline on an annotated dataset by 27% on the F1 score. |
GRAM: Fast Fine-tuning of Pre-trained Language Models for Content-based Collaborative Filtering (2022.naacl-main)
Copied to clipboard
| Challenge: | Content-based collaborative filtering (CF) predicts user-item interactions based on both items’ interaction history and item content information. |
| Approach: | They propose to combine item encodings with a multi-modality approach to improve training efficiency by 146x . |
| Outcome: | The proposed model improves training efficiency (up to 146x) on five datasets from two task domains of Knowledge Tracing and News Recommendation. |
Neural Data-to-Text Generation with LM-based Text Augmentation (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural data-to-text generation is a difficult task for many new applications because of a lack of training data. |
| Approach: | They propose a few-shot approach that augments the data available for training by generating new text samples based on replacing specific values by alternative ones from the same category and pairing the new text with data samples. |
| Outcome: | The proposed approach outperforms fully supervised sequence-to-sequence models with less than 10% of the training set on both datasets. |
Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural natural language generation and understanding models are data-hungry and require massive amounts of annotated data to be competitive. |
| Approach: | They propose a framework that automatically synthesizes weak labels from large-scale weakly-labeled data with a fine-tuned GPT-2 and adapts parameter updates to the models according to the estimated label-quality. |
| Outcome: | The proposed framework outperforms benchmark systems on the E2E and Weather datasets when 100% of the training data is used. |
Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialogue (P19-1)
Copied to clipboard
| Challenge: | Generating fluent natural language responses from structured semantic representations is a critical step in task-oriented conversational systems. |
| Approach: | They propose using tree-structured semantic representations for better discourse-level structuring and sentence-level planning and introduce a challenging dataset using this representation for the weather domain. |
| Outcome: | The proposed model improves discourse-level structuring and sentence-level planning on a weather domain and can be decoded to improve semantic correctness. |
ENT-DESC: Entity Description Generation by Exploring Knowledge Graph (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for knowledge-to-text generation use RDF triples or key-value pairs to generate a natural language description. |
| Approach: | They propose a large-scale dataset to facilitate the study of KG-to-text . they propose MGCN model architecture that incorporates aggregation methods to extract the rich graph information. |
| Outcome: | The proposed model can represent the original graph information more comprehensively and integrates multiple aggregation methods to extract the rich graph information. |
Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation (2023.acl-short)
Copied to clipboard
| Challenge: | End-to-end speech translation models have limited training data and are often inefficient due to the inconsistency of length and representation between speech and text. |
| Approach: | They find that the "modality gap" between speech and text data is not a major problem in E2E ST . they decouple the encoder to speech encoder and text encoder, and they find that there is a 'capacity gap' |
| Outcome: | The proposed model achieves 29.0 for en-de and 40.3 for fr on the MuST-C dataset. |
Dynamically Updating Event Representations for Temporal Relation Classification with Multi-category Learning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models with independent classifiers for each TLINK category hinder from using the whole data. |
| Approach: | They propose a temporal relation classification model that manages dynamic event representations across multiple TLINKs using multi-task learning to leverage the full size of data. |
| Outcome: | The proposed model outperforms state-of-the-art models and two strong transfer learning baselines on English and Japanese data. |
Multi-task Learning for Natural Language Generation in Task-Oriented Dialogue (D19-1)
Copied to clipboard
| Challenge: | Existing methods to generate natural language for task-oriented dialogues lack naturalness and variation in language. |
| Approach: | They propose a multi-task learning framework for natural language generation that explicitly targets for naturalness in generated responses via an unconditioned language model. |
| Outcome: | The proposed framework outperforms existing models across multiple datasets in the study of natural language generation. |
TempLM: Distilling Language Models into Template-Based Generators (2023.findings-acl)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) have greatly improved text generation, but they have also been known to produce unfaithful or inappropriate content. |
| Approach: | They propose a pretrained language model that is a template-based generator and uses it to generate a text. |
| Outcome: | The proposed model is more faithful than the original PLM and more fluent than prior template systems. |
SynTQA: Synergistic Table-based Question Answering via Mixture of Text-to-SQL and E2E TQA (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Text-to-SQL parsing and end-to end question answering have yet to be compared and their synergy remains unexplored. |
| Approach: | They propose a Synergistic Table-based Question Answering approach that integrates different models via answer selection. |
| Outcome: | The proposed approach improves on multiple benchmarks and on large scale datasets. |
An End-to-end Approach for Handling Unknown Slot Values in Dialogue State Tracking (P18-1)
Copied to clipboard
| Challenge: | a dialogue state tracker is a core component in most of today's spoken dialogue systems . slot-filling dialogues are composed of a predefined set of slots that need to be filled through the conversation . |
| Approach: | They propose an E2E architecture that extracts unknown slot values while still achieving state-of-the-art accuracy on the standard DSTC2 benchmark. |
| Outcome: | The proposed architecture achieves state-of-the-art accuracy on the DSTC2 benchmark while retaining predefined slot values. |
Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Training (2024.eacl-long)
Copied to clipboard
| Challenge: | End-to-end (E2E) spoken language understanding models are constrained by the cost of collecting speech-semantics pairs. |
| Approach: | They propose a model that learns E2E SLU without speech-semantics pairs . they propose cross-modal selective self-training (CMSST) to address imbalance and noise issues . |
| Outcome: | The proposed model learns E2E SLU without speech-semantics pairs . the proposed model requires the domains of speech-text and text-sensitization to match . |
Faithful Low-Resource Data-to-Text Generation through Cycle Training (2023.acl-long)
Copied to clipboard
| Challenge: | Methods to generate text from structured data have advanced significantly in recent years, but can fail to produce output faithful to the input data, especially on out-of-domain data. |
| Approach: | They evaluate the effectiveness of cycle training by using two models which are inverses of each other to generate text from structured data and one which generates the structured data from natural language text. |
| Outcome: | The proposed approach achieves nearly the same performance as fully supervised approaches on the WebNLG, E2E, WTQ, and WSQL datasets. |
GE2PE: Persian End-to-End Grapheme-to-Phoneme Conversion (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text-to-speech systems struggle to produce natural speech from grapheme sequences . Grapheme-to phoneme conversion (G2P) systems face limitations when dealing with Persian texts due to the complexity of Persian transcription. |
| Approach: | They propose to use phonetic information to enhance the input sequence for Persian translations. |
| Outcome: | The proposed model surpasses state-of-the-art models by 1.86% in word error rate and 3.42% in homograph disambiguation accuracy. |
Beyond Sentence-Level End-to-End Speech Translation: Context Helps (2021.acl-long)
Copied to clipboard
| Challenge: | Document-level contextual information has shown benefits to text-based machine translation, but whether and how it helps end-to-end speech translation is still under-studied. |
| Approach: | They propose a concatenation-based ST model with adaptive feature selection for computational efficiency. |
| Outcome: | The proposed model improves translation quality and robustness to (artificial) audio segmentation errors. |
WACO: Word-Aligned Contrastive Learning for Speech Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing ST methods perform poorly when only a limited amount of parallel data are available for training. |
| Approach: | They propose a Word-Aligned COntrastive learning method for low-resource speech-to-text translation that bridges word-level representations for both speech and text modalities via contrastive learning. |
| Outcome: | The proposed method outperforms the best baseline by 9+ BLEU points with only 1-hour parallel ST data. |
Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity (2020.coling-main)
Copied to clipboard
| Challenge: | End-to-end neural data-totext generation has faced challenges generalizing to new domains and generating semantically consistent text. |
| Approach: | They propose a neural data-to-text generation system that makes minimal assumptions about the data representation and target domain. |
| Outcome: | The proposed system achieves state of the art results on four major D2T datasets with better semantic fidelity than the state-of-the-art methods. |
Incremental Beam Manipulation for Natural Language Generation (2021.eacl-main)
Copied to clipboard
| Challenge: | a larger beam size can lead to deteriorating performance of natural language generation systems due to model errors . performance of NLG systems can plateau or even decrease when beam sizes larger than 10 are used . |
| Approach: | They propose to rerank the output of beam search to produce a good set of hypotheses . they propose incremental beam manipulation to discarded hypothese . |
| Outcome: | The proposed method outperforms a strong reranker on the E2E and WebNLG datasets while being on par with the existing method. |
FCGCL: Fine- and Coarse-Granularity Contrastive Learning for Speech Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to perform implicit knowledge transfer from machine translation to ST model are difficult because of the task complexity and data scarcity. |
| Approach: | They recommend a method which conducts explicit knowledge transfer from MT to ST model by fine and coarse granularity contrastive learning. |
| Outcome: | The proposed method improves the performance of the end-to-end speech translation model on all 8 languages. |
How to Make Neural Natural Language Generation as Reliable as Templates in Task-Oriented Dialogue (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural Natural Language Generation (NLG) systems are well known for their unreliability. |
| Approach: | They propose a data augmentation approach which restricts the output of a neural network and guarantees reliability. |
| Outcome: | The proposed approach scored 100% in semantic accuracy on the E2E NLG Challenge dataset, the same as a template system. |
Adaptive Feature Selection for End-to-End Speech Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end speech translation (E2E) models that directly maps audio to a foreign text are not efficient. |
| Approach: | They propose adaptive feature selection (AFS) for encoder-decoder based E2E ST. |
| Outcome: | The proposed model outperforms the existing model on LibriSpeech En-Fr and MuST-C with a BLEU score of 18.56. |
DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2023.eacl-main)
Copied to clipboard
| Challenge: | Pre-training/fine-tuning of pre-training models has become more expensive and resource-hungry. |
| Approach: | They propose a low-rank adaptation technique that trains LoRA blocks for a range of ranks instead of a single rank. |
| Outcome: | The proposed method trains LoRA blocks for a range of ranks instead of a single rank . it can train dynamic search-free models with DyLoRA at least 4 to 7 times faster than LoRA . |
Neural Pipeline for Zero-Shot Data-to-Text Generation (2022.acl-long)
Copied to clipboard
| Challenge: | In data-to-text generation, training on in-domain data leads to overfitting and repeating training data noise. |
| Approach: | They propose to train pretrained language models on general-domain text-based operations by transforming single-item descriptions with modules trained on ordering, aggregation, and paragraph compression. |
| Outcome: | The proposed approach enables D2T generation from RDF triples in zero-shot settings. |
Fast Randomized Low-Rank Adaptation of Pre-trained Language Models with PAC Regularization (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient way to fine-tune large language models (LLMs) but its memory overhead restricts it to scale up when the model size increases. |
| Approach: | They propose a low-rank adaptation method which decomposes model weight updates into a pair of low-ranked projection matrices and incorporates a regularizer to improve generalization. |
| Outcome: | The proposed method achieves better performance under few-shot settings and similar performance to the SOTA low-rank adaptation methods. |
Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Latent Synthesis is an efficient textual data utilization framework for end-to-end speech processing models . labeled speech data are scarcer and more expensive for collection compared to textual ones . |
| Approach: | They propose a textual data utilization framework for E2E speech processing models . they train a latent synthesizer to convert textual information into an intermediate latent representation . |
| Outcome: | The proposed framework improves on low-resource speech recognition and spoken language understanding tasks. |
Towards More Efficient Post-training via Fourier Domain Adapter Framework (2025.findings-emnlp)
Copied to clipboard
| Challenge: | FDA reparameterizes the core projection operation of the adapter module directly in the Fourier domain. |
| Approach: | They propose a framework that reparameterizes the core projection operation of the adapter module directly in the Fourier domain. |
| Outcome: | The proposed framework outperforms existing parameter-efficient fine-tuning methods on GLUE, E2E NLG, and instruction tuning benchmarks. |
End-to-End Entity Resolution and Question Answering Using Differentiable Knowledge Graphs (2021.emnlp-main)
Copied to clipboard
| Challenge: | End-to-end (E2E) trained models for question answering over knowledge graphs (KGQA) are effective, but training a weakly supervised dataset is difficult. |
| Approach: | They extend the boundaries of E2E learning for KGQA to include the training of an ER component. |
| Outcome: | The proposed model is fully differentiable thanks to a recent method for building differentiably KGs. |
Wav2Prompt: End-to-End Speech Prompt Learning and Task-based Fine-tuning for Text-based LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Text-based large language models (LLMs) can be applied to a wide range of tasks without being explicitly trained. |
| Approach: | They propose a method which integrates spoken input with a text-based large language model (LLM) it takes LLM token embeddings as training targets and utilises a continuous integrate-and-fire mechanism for explicit speech-text alignment. |
| Outcome: | The proposed model can be applied to speech translation, speech understanding and spoken-query-based question answering tasks. |
LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | ***LLaST*** is a framework for building high-performance Large Language model based Speech-to-text Translation systems. |
| Approach: | They propose a framework for building high-performance Large Language model based Speech-to-text Translation systems. |
| Outcome: | The proposed model outperforms the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. |
ULN: Towards Underspecified Vision-and-Language Navigation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-and-language navigation models are brittle to multi-level language underspecification. |
| Approach: | They propose to use multi-level underspecified instructions to guide agents . they propose to learn GSS for navigation agent to ground multi- level instructions . experimental results show existing VLN models are still brittle to multi-language underspecification . |
| Outcome: | Experimental results show that the proposed framework outperforms baselines on ULN by 10% relative success rate across all levels. |
Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translation (2024.acl-long)
Copied to clipboard
| Challenge: | LS-Transducer-SST dynamically decides when to emit translation tokens based on an Auto-regressive Integrate-and-Fire (AIF) mechanism. |
| Approach: | They propose a label-synchronous neural transducer for simultaneous speech translation which dynamically decides when to emit translation tokens based on an Auto-regressive Integrate-and-Fire mechanism. |
| Outcome: | The proposed model can predict translation tokens on monolingual text-only data or can be used in both decoding and training. |
Effective Integration of Text Diffusion and Pre-Trained Language Models with Linguistic Easy-First Schedule (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing noise schedules in text diffusion models do not take into account the linguistic differences among tokens, which violates the easy-first policy for text generation. |
| Approach: | They propose to use a pre-trained decoder to convert denoised embedding vectors into natural language instead of the widely used rounding operation. |
| Outcome: | The proposed model outperforms existing models on the E2E dataset and five controllable tasks on the discrete nature of text data. |
EfficientQAT: Efficient Quantization-Aware Training for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Quantization-aware training (QAT) is a low-bit training solution that requires substantial training resources. |
| Approach: | They propose an algorithm that reduces memory consumption by low-bit representations with minimal accuracy loss. |
| Outcome: | EfficientQAT achieves 2-bit Llama-2-70B model on single GPU in 41 hours . compared to previous methods, it obtains model with less than 3 points accuracy degradation . |
Learning Word Representations with Cross-Sentence Dependency for End-to-End Co-reference Resolution (D18-1)
Copied to clipboard
| Challenge: | Existing word embedding models generate word representations by running long short-term memory recurrent neural networks on each sentence of an input article or conversation separately. |
| Approach: | They propose a word embedding model that learns cross-sentence dependency . they use linear sentence linking and attentional sentence linking to learn cross-entry dependency based on context sentences . |
| Outcome: | The proposed model improves end-to-end co-reference resolution by taking knowledge from context sentences and the entire document. |
Where are we in Named Entity Recognition from Speech? (2020.lrec-1)
Copied to clipboard
| Challenge: | Named entity recognition is usually made through a pipeline process that consists of processing audio and applying a NER to the audio outputs. |
| Approach: | They propose an original 3-pass approach and explore the capability of an E2E system to do structured NER. |
| Outcome: | The proposed system performs better than the current pipeline approach. |
FalAI: A Dataset for End-to-end Spoken Language Understanding in a Low-Resource Scenario (2024.lrec-main)
Copied to clipboard
Andres Pineiro-Martin, Carmen Garcia-Mateo, Laura Docio-Fernandez, Maria del Carmen Lopez-Perez, Jose Gandarela-Rodriguez
| Challenge: | End-to-end (E2E) Spoken Language Understanding systems extract structured information from speech signals using a single model. |
| Approach: | They propose to use a dataset to extract structured information from speech signals . they define splits for noisy audio, hesitant audio and audio where sentence has changed . |
| Outcome: | The proposed model exploits acoustic information and avoids cascading errors . falAI dataset is the largest public SLU dataset in Galician and first to be obtained in low-resource scenario. |
Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence (2020.acl-main)
Copied to clipboard
| Challenge: | Recent neural attention models conflate all steps into a single end-to-end system and simplify training process. |
| Approach: | They propose to explicitly segment target text into fragment units and align them with their data correspondences. |
| Outcome: | The proposed model outperforms neural attention models on E2E and WebNLG benchmarks. |
Contrastive Error Attribution for Finetuned Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for error tracing do not detect faithfulness errors in NLG datasets. |
| Approach: | They propose a framework to identify and remove low-quality training instances that lead to undesirable outputs. |
| Outcome: | The proposed method outperforms existing methods for detecting faithfulness errors in NLG datasets. |
Generative Entity-to-Entity Stance Detection with Knowledge Graph Augmentation (2022.emnlp-main)
Copied to clipboard
| Challenge: | a new task is needed to understand the interaction between entities when inferring stances. |
| Approach: | They propose a task that primes models to identify entities in their canonical names and discern stances jointly. |
| Outcome: | The proposed model outperforms strong comparisons by large margins. |
MoKA:Parameter Efficiency Fine-Tuning via Mixture of Kronecker Product Adaption (2025.coling-main)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is one of the most popular PEFT methods . low-rank update mechanism of LoRA somewhat limits its ability to approximate full-parameter fine-tuning during training process. |
| Approach: | They propose a parameter-efficient fine-tuning framework that combines Kronecker product with the Mixture-of-Experts method to achieve parameter efficiency and better model performance. |
| Outcome: | The proposed framework outperforms existing methods on the GLUE benchmark and instruction tuning tasks for large language models. |
Corpus Generation for Voice Command in Smart Home and the Effect of Speech Synthesis on End-to-End SLU (2020.lrec-1)
Copied to clipboard
| Challenge: | Massive amounts of annotated data are often unavailable for novel tasks performed in real-world environments such as smart homes. |
| Approach: | They propose to use a synthetic semantically-annotated corpus of French commands for smart-home to train pipeline and end-to-end (E2E) SLU models. |
| Outcome: | The proposed model trains pipeline and end-to-end (E2E) SLU models on voice commands acquired in a real smart home. |
Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Mainstream of automatic speech recognition (ASR) has shifted from pipeline methods to end-to-end (E2E) methods. |
| Approach: | They propose to integrate a pre-trained speech representation model and a large language model (LLM) for automatic speech recognition in an end-to-end manner. |
| Outcome: | The proposed model achieves comparable performance to modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach. |
E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches decouple textual entity recognition and visual grounding, leading to error accumulation and suboptimal joint optimization. |
| Approach: | They propose a fully end-to-end generative framework that unifies recognition, semantic typing, visual grounding and implicit knowledge reasoning within a single multimodal large language model. |
| Outcome: | The proposed framework achieves highly competitive performance compared with state-of-the-art methods. |
Post-decoder Biasing for End-to-End Speech Recognition of Multi-turn Medical Interview (2024.lrec-main)
Copied to clipboard
| Challenge: | End-to-end (E2E) models are replacing hybrid models for automatic speech recognition tasks. |
| Approach: | They propose a method to optimize E2E models for automatic speech recognition . they propose MED-IT, a multi-turn consultation speech dataset . |
| Outcome: | The proposed method improves on subsets of rare words appearing in training speech. |
Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction (2026.acl-long)
Copied to clipboard
Advait Gosai, Tyler Vuong, Utkarsh Tyagi, Steven Li, Wenjia You, Miheer Bavare, Arda Uçar, Zhongwang Fang, Brian Jang, Bing Liu, Yunzhong He
| Challenge: | End-to-end (E2E) spoken dialogue systems are replacing cascaded pipelines for voice-based human-AI interaction. Existing benchmarks evaluate these systems on synthetic speech and single-turn tasks, leaving multi-turn conversational ability underexplored. |
| Approach: | They propose an open-source benchmark to evaluate spoken dialogue systems under natural multi-turn interaction patterns. |
| Outcome: | The proposed model fails on the highest-performing model with 54.65% pass rate. |