Papers by Supriti Sinhamahapatra
Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current automatic speech recognition systems rely on only audio information, ignoring multi-modal context. |
| Approach: | They propose to integrate visual context into existing automatic speech recognition systems to integrate presentation slides with multi-modal information. |
| Outcome: | The proposed model reduces word error rate by approximately 34% across all words and 35% for domain-specific terms compared to baseline model. |
How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent advances in foundation models have sparked growing interest in expanding their text processing capabilities to speech. |
| Approach: | They analyze the model activations from semantically equivalent sentences across languages in the text and speech modalities and examine how text and spoken are represented in recent multimodal foundation models. |
| Outcome: | The proposed models exhibit cross-lingual differences, but are not explicitly trained for modality-agnostic representations. |