ConEC: Earnings Call Dataset with Real-world Contexts for Benchmarking Contextual Speech Recognition (2024.lrec-main)
Copied to clipboard
Ruizhe Huang, Mahsa Yarmohammadi, Jan Trmal, Jing Liu, Desh Raj, Leibny Paola Garcia, Alexei V. Ivanov, Patrick Ehlen, Mingzhi Yu, Dan Povey, Sanjeev Khudanpur
| Challenge: | Existing work on contextual speech recognition (ASR) systems focuses on recognizing words that are not frequently seen in training data, such as rare words, but word error rate on rare words remains over 20%. |
| Approach: | They propose to use public-domain earnings calls and supplementary materials to evaluate contextual ASR approaches grounded on real-world applications. |
| Outcome: | The proposed frameworks are noisier than artificially synthesized contexts that contain the ground truth, yet still make great room for future improvement of contextual ASR technology. |
Similar Papers
Benchmarking Contextual and Paralinguistic Reasoning in Speech-LLMs: A Case Study with In-the-Wild Data (2025.findings-emnlp)
Copied to clipboard
Qiongqiong Wang, Hardik Bhupendra Sailor, Tianchi Liu, Wenyu Zhang, Muhammad Huzaifah, Nattadaporn Lertcheva, Shuo Sun, Nancy F. Chen, Jinyang Wu, AiTi Aw
| Challenge: | Recent speech-LLMs have shown impressive performance in tasks like transcription and translation, yet they remain limited in understanding the paralinguistic aspects of speech crucial for social and emotional intelligence. |
| Approach: | They propose a benchmark for evaluating speech-LLMs on contextual paralinguistic reasoning . the benchmark includes curated question answering datasets requiring both linguistic and empathetic understanding . |
| Outcome: | The proposed benchmark reveals a key gap in existing evaluations and offers insights into building more context-aware and emotionally intelligent LLMs. |
Evaluation of Off-the-shelf Speech Recognizers Across Diverse Dialogue Domains (2020.lrec-1)
Copied to clipboard
| Challenge: | a recent study evaluated off-the-shelf automatic speech recognition systems . current state-of-the art systems perform poorly in domains that require special vocabulary and language models . |
| Approach: | They evaluate off-the-shelf automatic speech recognition systems across different dialogue domains . they use data collected from deployed spoken dialogue systems and human-human conversations . |
| Outcome: | The evaluation is aimed at non-experts with limited experience in speech recognition . the results show that the performance of each speech recognizer can vary significantly depending on the domain . |
CEASR: A Corpus for Evaluating Automatic Speech Recognition (2020.lrec-1)
Copied to clipboard
Malgorzata Anna Ulasik, Manuela Hürlimann, Fabian Germann, Esin Gedik, Fernando Benites, Mark Cieliebak
| Challenge: | Automatic Speech Recognition (ASR) systems are increasingly needed for research and practical applications. |
| Approach: | They propose to use public speech corpora to evaluate the quality of automatic speech recognition (ASR) they calculate an average Word Error Rate (WER) per corpus, per system and per corpor-system pair . |
| Outcome: | The proposed corpus evaluates the quality of automatic speech recognition systems using public speech corpora and transcripts generated by state-of-the-art systems. |
Effective Performance Measurement: Challenges and Opportunities in KPI Extraction from Earnings Calls (2026.acl-industry)
Copied to clipboard
Rasmus T. Aavang, Rasmus Tjalk-Bøggild, Alexandre Iolov, Giovanni Rizzi, Mike Zhang, Johannes Bjerva
| Challenge: | Earnings calls are a key source of financial information about public companies. extracting information from earnings calls is difficult. |
| Approach: | They propose to use LLMs to perform open-ended extraction from unstructured call transcripts to provide a baseline for this valuable domain through the consistent tracking of emergent KPIs. |
| Outcome: | The proposed method provides a baseline for this valuable domain through the consistent tracking of emergent KPIs. |
Evaluating Open-Source ASR Systems: Performance Across Diverse Audio Conditions and Error Correction Methods (2025.coling-main)
Copied to clipboard
| Challenge: | Automated speech recognition (ASR) systems are able to transcribe spontaneous human conversations with high accuracy. |
| Approach: | They evaluate the accuracy of open source automatic speech recognition systems across conversational speech datasets and explore the potential of ASR ensembling and post-ASR correction methods to improve transcription accuracy. |
| Outcome: | The proposed methods highlight the need for robust error correction techniques and address demographic biases to enhance ASR performance and inclusivity. |
Discourse on ASR Measurement: Introducing the ARPOCA Assessment Tool (2022.acl-srw)
Copied to clipboard
| Challenge: | Automated speech recognition (ASR) models are based on a corpus of audio recordings, but are often small or nonexistent for less common languages and dialects. |
| Approach: | This research proposal will develop a semi-automatic acoustic features extraction system that integrates phonetic transcripts with pronunciation dictionaries. |
| Outcome: | The proposed system will be used to improve language recognition and model feedback in less common languages and dialects. |
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
Decoding the Market’s Pulse: Context-Enriched Agentic Retrieval Augmented Generation for Predicting Post-Earnings Price Shocks (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for forecasting large stock price movements after corporate earnings calls are prone to **narrative bias** Existing approaches lack temporal-causal reasoning and are unable to predict large stock prices. |
| Approach: | They propose a retrieval-augmented framework that deploys a team of cooperative LLM agents . they retrieve structured evidence from a Causal-Temporal Knowledge Graph built from financial statements and earnings calls . |
| Outcome: | The proposed framework outperforms larger LLMs and fine-tuned models in macro-F1, MCC, and Sharpe for the same forecasting horizon. |
LingVarBench: Benchmarking LLMs on Entity Recognitions and Linguistic Verbalization Patterns in Phone-Call Transcripts (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing methods degrade under disfluencies, interruptions, and speaker overlap, yet large real-call corpora are rarely shareable. |
| Approach: | They propose a benchmark and semantic synthetic data generation pipeline that generates linguistically varied training data via (1) LLM-sampled entity values, (2) curated linguistic verbalization patterns covering diverse disfluencies and entity-specific readout styles, and (3) a value–transcript consistency filter. |
| Outcome: | The proposed pipeline outperforms zero-shot baselines and matches or closely approaches human-tuned prompts on real customer transcripts. |
Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current automatic speech recognition systems rely on only audio information, ignoring multi-modal context. |
| Approach: | They propose to integrate visual context into existing automatic speech recognition systems to integrate presentation slides with multi-modal information. |
| Outcome: | The proposed model reduces word error rate by approximately 34% across all words and 35% for domain-specific terms compared to baseline model. |