Papers by Nicolai Jacobsen
MultiQT: Multimodal learning for real-time question tracking in speech (2020.acl-main)
Copied to clipboard
Jakob D. Havtorn, Jan Latko, Joakim Edin, Lars Maaløe, Lasse Borgholt, Lorenzo Belgrano, Nicolai Jacobsen, Regitze Sdun, Željko Agić
| Challenge: | a novel multimodal approach to real-time sequence labeling in speech is proposed . the model treats speech and its own textual representation as two separate modalities . |
| Approach: | They propose a multimodal approach to real-time sequence labeling in speech . they use audio and transcription to jointly learn from a phone call . results show similar pattern of improvements with multimodal learning . |
| Outcome: | The proposed model shows significant gains under adverse noise and limited training data compared to text or audio only under adverse conditions and generalizes to medical symptoms detection. |