Papers by Karen Livescu
Open-Domain Sign Language Translation Learned from Online Video (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on sign language translation has focused mainly on data collected in controlled environments or domains, which limits its applicability to real-world settings. |
| Approach: | They propose to use sign search as a pretext task and fusion of mouthing and handshape features to improve sign language translation in real-world settings. |
| Outcome: | The proposed techniques produce consistent and large improvements over baseline models based on prior work. |
UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions (2024.naacl-long)
Copied to clipboard
Siddhant Arora, Hayato Futami, Jee-weon Jung, Yifan Peng, Roshan Sharma, Yosuke Kashiwagi, Emiru Tsunoo, Karen Livescu, Shinji Watanabe
| Challenge: | Recent studies leverage large language models with multi-tasking capabilities, using natural language prompts to guide the model’s behavior and surpassing performance of task-specific models. |
| Approach: | They adapt a pre-trained automatic speech recognition model to additional tasks using single-token task specifiers. |
| Outcome: | The proposed model can generalize to new datasets and languages for seen task types. |
Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing (2024.acl-long)
Copied to clipboard
| Challenge: | Recent work has proposed a new task of textless speech constituency parsing that uses textless parsers to parse spoken word boundaries over automatically recognized spoken word borders. |
| Approach: | They propose a metric that compares a constituency parse tree over spoken word boundaries with a ground-truth parser tree over written words. |
| Outcome: | The proposed metric shows higher tolerance to syntactically plausible parses than PARSEVAL. |
PeTra: A Sparsely Supervised Memory Model for People Tracking (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for coreference resolution scale quadratically with length of text, and are cognitively implausible. |
| Approach: | They propose a memory-augmented neural network called PeTra that tracks entities in its memory slots. |
| Outcome: | The proposed model outperforms a previous memory model while maintaining strong performance. |
Parsing Speech: a Neural Approach to Integrating Lexical and Acoustic-Prosodic Information (N18-1)
Copied to clipboard
| Challenge: | acoustic signals provide cues that help listeners disambiguate difficult parses . speech carries useful extra information associated with prosodic structure . |
| Approach: | They propose a model that integrates transcribed text and acoustic-prosodic features into a neural network that accepts text and prosodic feature. |
| Outcome: | The proposed model improves parse and disfluency detection scores over a strong text-only baseline. |
Visually Grounded Neural Syntax Acquisition (P19-1)
Copied to clipboard
| Challenge: | a visually grounded neural syntax learner is an approach for learning syntactic representations without any supervision. |
| Approach: | They propose a visually grounded neural syntax learner that acquires syntax by looking at images and reading captions. |
| Outcome: | The proposed model outperforms unsupervised approaches on the MSCOCO data set . it is more stable with choice of initialization and amount of training data, the authors show . |
Pre-training on high-resource speech recognition improves low-resource speech-to-text translation (N19-1)
Copied to clipboard
| Challenge: | Pre-training on high-resource automatic speech recognition (ASR) tasks improves ST performance even when source language is low-resourced. |
| Approach: | They propose a method to improve direct speech-to-text translation when source language is low-resource . they pre-train model on high-res automatic speech recognition task and fine-tune parameters for ST . |
| Outcome: | The proposed approach improves Spanish English ST even when the source language is low-resource . the pre-trained encoder accounts for most of the improvement, the authors show . |
SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work on sign language video processing focuses on the face, hands and body posture of the signer. |
| Approach: | They propose to learn the handshapes and rich facial expressions of sign languages in a self-supervised fashion by learning from individual frames rather than video sequences. |
| Outcome: | The proposed model is more efficient than previous work on sign language pre-training. |
Towards Robust Speech Representation Learning for Thousands of Languages (2024.emnlp-main)
Copied to clipboard
William Chen, Wangyou Zhang, Yifan Peng, Xinjian Li, Jinchuan Tian, Jiatong Shi, Xuankai Chang, Soumi Maiti, Karen Livescu, Shinji Watanabe
| Challenge: | XEUS is a cross-lingual encoder for universal speech that can be trained on 1 million hours of data across 4057 languages. |
| Approach: | They propose a Cross-lingual Encoder for Universal Speech that can be trained on 1 million hours of data across 4057 languages and a newly created corpus of 7400+ hours from 4057 . |
| Outcome: | The proposed model outperforms state-of-the-art models on several benchmarks and outperfies MMS 1B and w2v-BERT 2.0 v2 by 0.8% and 4.4% respectively. |
Self-supervised Representation Learning for Speech Processing (2022.naacl-tutorials)
Copied to clipboard
Hung-yi Lee, Abdelrahman Mohamed, Shinji Watanabe, Tara Sainath, Karen Livescu, Shang-Wen Li, Shu-wen Yang, Katrin Kirchhoff
| Challenge: | Self-supervised representation learning (SSL) uses proxy supervised learning tasks to obtain training data from unlabeled corpora. |
| Approach: | They propose to survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing to scale up current machine learning technologies. |
| Outcome: | The proposed tutorial is highly relevant to the special theme of ACL about language diversity. |
Baked-in State Probing (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows language models trained on form can capture aspects of meaning without explicit state supervision. |
| Approach: | They propose to use probing to "bake" state knowledge into language models . they propose to probe for underlying world state knowledge via text prompts . |
| Outcome: | The proposed methods show that language models trained on form can capture the world state without state supervision. |
On the Role of Supervision in Unsupervised Constituency Parsing (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on unsupervised constituency parsing uses labeled examples for tuning . a few-shot parser with labeles can outperform other approaches by a significant margin . |
| Approach: | They propose to use as few labeled examples as possible for model development . they propose to train existing models on the same labeles they access . |
| Outcome: | The proposed model outperforms other models on the WSJ development set by a significant margin . the proposed model can be further improved by augmentation and self-training . |
Searching for fingerspelled content in American Sign Language (2022.acl-long)
Copied to clipboard
| Challenge: | Sign language processing is gaining interest in the computer vision and natural language processing communities. |
| Approach: | They propose a model that jointly detects fingerspelling and matches it to a text sequence. |
| Outcome: | The proposed model outperforms baseline methods on a large public dataset of ASL fingerspelling in the wild. |
Variational Sequential Labelers for Semi-Supervised Learning (D18-1)
Copied to clipboard
| Challenge: | a family of multitask variational methods for semi-supervised sequence labeling is currently unclear how to use them in the context of sequence labelling. |
| Approach: | They propose a family of multitask variational methods for semi-supervised sequence labeling using latent variables and a discriminative labeler. |
| Outcome: | The proposed models outperform standard sequential baselines on 8 sequence labeling datasets and improve further with unlabeled data. |
Substructure Distribution Projection for Zero-Shot Cross-Lingual Dependency Parsing (2022.acl-long)
Copied to clipboard
| Challenge: | Substructure distribution projection (SubDP) projects a distribution over structures in one domain to another, using substructure distributions as silver labels. |
| Approach: | They evaluate substructure distribution projection technique on zero shot cross-lingual dependency parsing using dependency arcs as substructures. |
| Outcome: | The proposed technique achieves better unlabeled attachment score than previous work on eight target languages and best labeled attachment scores on six languages. |
Discrete Latent Variable Representations for Low-Resource Text Classification (2020.acl-main)
Copied to clipboard
| Challenge: | Several approaches to learning discrete latent variable models for text are available. |
| Approach: | They compare several approaches to learning discrete latent variable models for text in the case where exact marginalization over these variables is intractable. |
| Outcome: | The learned models outperform the previous best models in low-resource settings while learning significantly more compressed representations. |
On the Use of External Data for Spoken Named Entity Recognition (2022.naacl-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) tasks require large labeled datasets to perform . compared to prior work, relative improvements in F1 of up to 16% are found . |
| Approach: | They propose to use self-training, knowledge distillation, and transfer learning to learn SLU models . they compare pipeline and pipeline approaches to find out how to use external data . |
| Outcome: | The proposed models improve performance beyond pre-trained models in resource-constrained settings . the best baseline model is a pipeline approach, while the best performance is achieved by an E2E model. |
SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks (2023.acl-long)
Copied to clipboard
Suwon Shon, Siddhant Arora, Chyi-Jiunn Lin, Ankita Pasad, Felix Wu, Roshan S Sharma, Wei-Lun Wu, Hung-yi Lee, Karen Livescu, Shinji Watanabe
| Challenge: | Spoken language understanding (SLU) tasks have received little attention and resources compared to lower-level tasks like speech and speaker recognition. |
| Approach: | They propose annotated SLU benchmark tasks based on freely available speech data to complement existing benchmarks and address gaps in the evaluation landscape. |
| Outcome: | The proposed benchmarks complement existing benchmarks and address gaps in the evaluation landscape. |
On the Evaluation of Speech Foundation Models for Spoken Language Understanding (2024.findings-acl)
Copied to clipboard
Siddhant Arora, Ankita Pasad, Chung-Ming Chien, Jionghao Han, Roshan Sharma, Jee-weon Jung, Hira Dhamyal, William Chen, Suwon Shon, Hung-yi Lee, Karen Livescu, Shinji Watanabe
| Challenge: | Spoken language understanding evaluation (SLUE) benchmarks are used to benchmark complex spoken language understanding tasks on natural speech. |
| Approach: | They propose a set of benchmark tasks to evaluate spoken language understanding on natural speech . they use pre-trained speech foundation models to evaluate the utility of different SFMs . |
| Outcome: | The proposed framework outperforms pre-trained speech foundation models on natural speech . the proposed framework also outperformed self-supervised SFMs on the sequence generation tasks . |
SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for sign language processing have relied on task-specific models, limiting the potential for transfer learning across tasks. |
| Approach: | They propose a self-supervised contextual representation model that adapts masked token prediction objectives to multi-stream visual sign language input. |
| Outcome: | The proposed model adapts masked token prediction objectives to multi-stream visual sign language input, learning to predict multiple targets corresponding to clustered hand, face, and body pose streams. |
Cross-Modal Taxonomic Generalization in (Vision-) Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that language models learn from surface form to learn from more grounded evidence. |
| Approach: | They propose to use a vision-language model to learn hypernyms from images . they find that the model can recover this knowledge and generalize even when there is no hypernomia in the image. |
| Outcome: | The proposed model can recover this knowledge and generalize even when the model receives no evidence of hypernyms during training. |
Toward Joint Language Modeling for Speech Units and Text (2023.findings-emnlp)
Copied to clipboard
Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau, Alexei Baevski, Michael Auli
| Challenge: | Speech and text are two major forms of human language and little effort has been made to model them together. |
| Approach: | They propose to combine speech and text models to create mixed speech-text data by using different tokenizers and automatic metrics to evaluate how well the model mixes speech and texts. |
| Outcome: | The proposed model improves over a speech-only baseline and shows zero-shot cross-modal transferability. |
Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks (2020.emnlp-main)
Copied to clipboard
| Challenge: | Current models for document coreference resolution have large memory requirements and quadratic runtime in document length. |
| Approach: | They propose a memory-augmented neural network that tracks only a small number of entities at a time. |
| Outcome: | The proposed model outperforms existing models on OntoNotes and LitBank in memory management and memory management. |
Substructure Substitution: Structured Data Augmentation for NLP (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing work focuses on word-level manipulation or global sequence-to-sequence style generation. |
| Approach: | They propose a family of data augmentation methods that generalize prior methods by substituting substructures with others having the same label. |
| Outcome: | The proposed methods can be applied to many structured NLP tasks such as part-of-speech tagging and parsing. |