Papers by Chang Yoo
Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language Recognition (2023.findings-acl)
Copied to clipboard
Liming Wang, Junrui Ni, Heting Gao, Jialu Li, Kai Chieh Chang, Xulin Fan, Junkai Wu, Mark Hasegawa-Johnson, Chang Yoo
| Challenge: | Existing supervised sign language recognition systems rely on well-annotated data . instead, an unsupervised speech-to-sign language recognition system learns to translate between spoken and sign languages by observing only non-parallel speech and sign-language corpora. |
| Approach: | They propose an unsupervised speech-to-sign language recognition system that can translate between spoken and sign languages by observing only non-parallel speech and sign-language corpora. |
| Outcome: | The proposed approach outperforms baseline models on sign language corpora by 50% . the proposed approach is available at https://github.com/cactuswiththoughts/UnsupSpeech2Sign.git . |
Self-supervised Semantic-driven Phoneme Discovery for Zero-resource Speech Recognition (2022.acl-long)
Copied to clipboard
| Challenge: | a new approach for self-supervised speech representation learning is proposed . a phoneme inventory learning model is based on a discrete representation of speech . |
| Approach: | They propose a neural discrete representation learning model for self-supervised phoneme inventory learning with raw speech and word labels. |
| Outcome: | The proposed model learns better phoneme-level representations and lowers error rates on TIMIT and Mboshi benchmarks than previous state-of-the-art models. |
Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | Experiments with AudioCaps, Clotho, and MECAT show that OEA achieves comparable text-to-text retrieval performance to state-of-the-art M2D-CLAP. |
| Approach: | They propose a retrieval-oriented encoder leveraging multimodal LLMs with native audio understanding that allows users to express their queries in five different ways. |
| Outcome: | Experiments on AudioCaps, Clotho, and MECAT show that OEA achieves comparable text-to-audio retrieval performance to state-of-the-art M2D-CLAP while demonstrating clear advantages in two critical areas. |
Worldly Wise (WoW) - Cross-Lingual Knowledge Fusion for Fact-based Visual Spoken-Question Answering (2021.naacl-main)
Copied to clipboard
| Challenge: | Question-Answering has long been of interest, but its accessibility to users through a speech interface and its support to multiple languages have not been addressed in prior studies. |
| Approach: | They propose a task and a synthetically-generated dataset to do Fact-based Visual Spoken-Question Answering (FVSQA) the task requires a system to retrieve an entity from Knowledge Graphs (KGs) the question is spoken rather than typed. |
| Outcome: | The proposed task performs at same levels of accuracy across 3 languages, including English, Hindi, and Turkish. |
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (2021.emnlp-main)
Copied to clipboard
Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Jeon Dong Hyeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, Suk Hyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, Na-Hyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung-Woo Ha, Woomyoung Park, Nako Sung
| Challenge: | GPT-3 has been used to train large-scale language models on hundreds of billion scale data. |
| Approach: | They propose a Korean variant of GPT-3 that uses Korean tokens to train in-context models. |
| Outcome: | The proposed method shows state-of-the-art zero-shot and few-shot learning on downstream tasks in Korean. |
TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback (2024.findings-acl)
Copied to clipboard
Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung-Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, Chang Yoo
| Challenge: | Existing approaches to provide token-level rewards fail to account for varying degrees of preference inherent to each token. |
| Approach: | They propose a reward model that uses a discriminator to assign token-based continuous rewards to each token considering the context. |
| Outcome: | Extensive experiments show that the proposed reward model improves on open-ended language generation benchmarks. |
A Theory of Unsupervised Speech Recognition (2023.acl-long)
Copied to clipboard
| Challenge: | Unsupervised speech recognition (ASR) is the problem of learning automatic speech recognition systems from unpaired speech-only and text-only corpora. |
| Approach: | They propose a general theoretical framework to study the properties of pasted macro ‘ASRU’/ systems based on random matrix theory and the theory of neural tangent kernels. |
| Outcome: | The proposed framework proves various learnability conditions and sample complexity bounds on synthetic languages with three classes of transition graphs. |
Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances (2022.naacl-main)
Copied to clipboard
| Challenge: | In this paper, we consider mimicking fictional characters as a promising direction for building engaging conversation models. |
| Approach: | They propose a task where only a few utterances of each fictional character are available to generate responses mimicking them. |
| Outcome: | The proposed method generates responses better reflecting the style of fictional characters than baseline methods. |
Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing video-grounded dialogue systems suffer from text hallucination problem due to learning spurious correlations from the fact that answer sentences in the dataset usually include the words of input texts. |
| Approach: | They propose to decode an answer sentence to a question using video and dialogue contexts. |
| Outcome: | The proposed framework shows that it generates adequate conversational responses to the queries of humans while following up on video and dialogue context. |
HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems are competent only to incorporate information in the video and text and tend to struggle in extracting the necessary information from the audio when generating appropriate responses to the question. |
| Approach: | They propose to perform sensible listening by selectively attending to audio whenever the question requires it. |
| Outcome: | The proposed framework enhances the accuracy and audibility of VGD systems in a model-agnostic manner. |
Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that capture multiple modalities with a single input length are unable to handle this computational burden. |
| Approach: | They propose a query-based cross-modal projector that compresses visual tokens based on input through the cross-attention mechanism. |
| Outcome: | The proposed projector reduces the need for manually designing the 2D scan order of original image features when converting them into an input sequence for Mamba LLMs. |
Efficient Latent Variable Modeling for Knowledge-Grounded Dialogue Generation (2023.findings-emnlp)
Copied to clipboard
Gunsoo Han, Daejin Jo, Daniel Nam, Eunseop Yoon, Taehwan Kwon, Seungeun Rho, Kyoung-Woon On, Chang Yoo, Sungwoong Kim
| Challenge: | Existing knowledge-grounded dialogue generation algorithms require annotated knowledge to generate a response grounded on the retrieved knowledge. |
| Approach: | They propose an efficient algorithm for latent variable modeling that leverages large amount of dialogue data. |
| Outcome: | The proposed algorithm outperforms the supervised learning algorithm on knowledge-grounded dialogue datasets while maintaining efficiency and scalability. |
SMSMix: Sense-Maintained Sentence Mixup for Word Sense Disambiguation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is an NLP task aimed at determining the correct sense of a word in a sentence from discrete sense choices. |
| Approach: | They propose a word-level mixup method that maintains the sense of a target word by increasing the frequency of least frequent senses during training to reduce the distributional bias of senses. |
| Outcome: | The proposed method shows high accuracy on the most frequent sense (MFS) and low accuracy on least frequent senses (LFS). |
Variational Hierarchical Dialog Autoencoder for Dialog State Tracking Data Augmentation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that generative data augmentation, where synthetic samples generated from deep generative models complement the training dataset, benefit NLP tasks. |
| Approach: | They propose a Variational Hierarchical Dialog Autoencoder for modeling the complete aspects of goal-oriented dialogs using inter-connected latent variables and learns to generate coherent dialogs from the latent spaces. |
| Outcome: | The proposed model outperforms previous strong baselines on dialog response generation and user simulation tasks. |
INTapt: Information-Theoretic Adversarial Prompt Tuning for Enhanced Non-Native Speech Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to improve ASR performance with pre-trained models require updating the pre-training model weights. |
| Approach: | They propose a method that uses prompts concatenated to the original input to re-modulate attention of the pre-trained model. |
| Outcome: | The proposed model improves the performance of L2 English and increases similarity between L2 and L1 accents. |