Papers by Mark Hasegawa-Johnson
Finding Spoken Identifications: Using GPT-4 Annotation for an Efficient and Fast Dataset Creation Pipeline (2024.lrec-main)
Copied to clipboard
Maliha Jahan, Helin Wang, Thomas Thebaud, Yinglun Sun, Giang Ha Le, Zsuzsanna Fagyal, Odette Scharenborg, Mark Hasegawa-Johnson, Laureano Moro Velazquez, Najim Dehak
| Challenge: | Existing datasets that are limited to a few dialects, ethnicities, and age groups are not annotated considering these factors. |
| Approach: | They propose a semi-automated dataset creation pipeline that leverages large language models to perform two complex annotation tasks using human annotations as ground truths. |
| Outcome: | The proposed pipeline reduces time required for the filtering and tagging tasks while losing no important information. |
Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language Recognition (2023.findings-acl)
Copied to clipboard
Liming Wang, Junrui Ni, Heting Gao, Jialu Li, Kai Chieh Chang, Xulin Fan, Junkai Wu, Mark Hasegawa-Johnson, Chang Yoo
| Challenge: | Existing supervised sign language recognition systems rely on well-annotated data . instead, an unsupervised speech-to-sign language recognition system learns to translate between spoken and sign languages by observing only non-parallel speech and sign-language corpora. |
| Approach: | They propose an unsupervised speech-to-sign language recognition system that can translate between spoken and sign languages by observing only non-parallel speech and sign-language corpora. |
| Outcome: | The proposed approach outperforms baseline models on sign language corpora by 50% . the proposed approach is available at https://github.com/cactuswiththoughts/UnsupSpeech2Sign.git . |
Self-supervised Semantic-driven Phoneme Discovery for Zero-resource Speech Recognition (2022.acl-long)
Copied to clipboard
| Challenge: | a new approach for self-supervised speech representation learning is proposed . a phoneme inventory learning model is based on a discrete representation of speech . |
| Approach: | They propose a neural discrete representation learning model for self-supervised phoneme inventory learning with raw speech and word labels. |
| Outcome: | The proposed model learns better phoneme-level representations and lowers error rates on TIMIT and Mboshi benchmarks than previous state-of-the-art models. |
Worldly Wise (WoW) - Cross-Lingual Knowledge Fusion for Fact-based Visual Spoken-Question Answering (2021.naacl-main)
Copied to clipboard
| Challenge: | Question-Answering has long been of interest, but its accessibility to users through a speech interface and its support to multiple languages have not been addressed in prior studies. |
| Approach: | They propose a task and a synthetically-generated dataset to do Fact-based Visual Spoken-Question Answering (FVSQA) the task requires a system to retrieve an entity from Knowledge Graphs (KGs) the question is spoken rather than typed. |
| Outcome: | The proposed task performs at same levels of accuracy across 3 languages, including English, Hindi, and Turkish. |
TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback (2024.findings-acl)
Copied to clipboard
Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung-Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, Chang Yoo
| Challenge: | Existing approaches to provide token-level rewards fail to account for varying degrees of preference inherent to each token. |
| Approach: | They propose a reward model that uses a discriminator to assign token-based continuous rewards to each token considering the context. |
| Outcome: | Extensive experiments show that the proposed reward model improves on open-ended language generation benchmarks. |
A Theory of Unsupervised Speech Recognition (2023.acl-long)
Copied to clipboard
| Challenge: | Unsupervised speech recognition (ASR) is the problem of learning automatic speech recognition systems from unpaired speech-only and text-only corpora. |
| Approach: | They propose a general theoretical framework to study the properties of pasted macro ‘ASRU’/ systems based on random matrix theory and the theory of neural tangent kernels. |
| Outcome: | The proposed framework proves various learnability conditions and sample complexity bounds on synthetic languages with three classes of transition graphs. |
Syn2Vec: Synset Colexification Graphs for Lexical Semantic Similarity (2022.naacl-main)
Copied to clipboard
| Challenge: | In this paper we examine patterns of colexification as an aspect of lexical-semantic organization, and compare several approaches to build large scale graphs across 499 world languages. |
| Approach: | They propose to use patterns of colexification as an aspect of lexical-semantic organization to build large scale synset graphs across a typologically diverse set of 499 world languages. |
| Outcome: | The proposed models are evaluated against human judgments on a semantic similarity task for nine languages. |
Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that capture multiple modalities with a single input length are unable to handle this computational burden. |
| Approach: | They propose a query-based cross-modal projector that compresses visual tokens based on input through the cross-attention mechanism. |
| Outcome: | The proposed projector reduces the need for manually designing the 2D scan order of original image features when converting them into an input sequence for Mamba LLMs. |
SMSMix: Sense-Maintained Sentence Mixup for Word Sense Disambiguation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is an NLP task aimed at determining the correct sense of a word in a sentence from discrete sense choices. |
| Approach: | They propose a word-level mixup method that maintains the sense of a target word by increasing the frequency of least frequent senses during training to reduce the distributional bias of senses. |
| Outcome: | The proposed method shows high accuracy on the most frequent sense (MFS) and low accuracy on least frequent senses (LFS). |
INTapt: Information-Theoretic Adversarial Prompt Tuning for Enhanced Non-Native Speech Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to improve ASR performance with pre-trained models require updating the pre-training model weights. |
| Approach: | They propose a method that uses prompts concatenated to the original input to re-modulate attention of the pre-trained model. |
| Outcome: | The proposed model improves the performance of L2 English and increases similarity between L2 and L1 accents. |