Papers by Akshay Raghavan
WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher Learning (2025.acl-long)
Copied to clipboard
Rajath Rao, Adithya V Ganesan, Oscar Kjell, Jonah Luby, Akshay Raghavan, Scott M. Feltman, Whitney Ringwald, Ryan L. Boyd, Benjamin J. Luft, Camilo J. Ruggero, Neville Ryant, Roman Kotov, H. Schwartz
| Challenge: | Current speech encoding pipelines rely on an additional text-based LM to get robust representations of human communication, even though speech-to-text models often have a LM within. |
| Approach: | They propose to align Whisper's latent space with semantic representations from a text autoencoder and lexically derived embeddings of basic psychological dimensions: emotion and personality. |
| Outcome: | The proposed approach surpasses current speech encoders over self-supervised affective tasks and downstream psychological tasks, achieving an error reduction of 73.4% and 83.8%, respectively. |