Symbolic Inductive Bias for Visually Grounded Learning of Spoken Language (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to processing spoken language are to first automatically transcribe it into text, but there is an alternative: end-to-end learning. |
| Approach: | They propose to use multitask learning to exploit existing transcribed speech within the end-to-end setting by matching spoken captions with corresponding images, speech with text, and text with images. |
| Outcome: | The proposed model improves image retrieval performance compared to training the speech/image task in isolation. |
Similar Papers
Textual Supervision for Visually Grounded Spoken Language Understanding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | a new approach to spoken language understanding extracts semantic information directly from speech without relying on transcriptions. |
| Approach: | They propose to use textual supervision to train visually-grounded models of spoken language understanding without relying on transcriptions. |
| Outcome: | The proposed model improves when enough text is available, the study shows . compared with pipeline-based models, the pipeline approach performs better when enough data is available . |
Grounded PCFG Induction with Images (2020.aacl-main)
Copied to clipboard
| Challenge: | Recent work in unsupervised parsing has tried to incorporate visual information into learning, but results suggest that these models need linguistic bias to compete against models that only rely on text. |
| Approach: | They propose to use visual information from images for labeled parsing and compare them to existing models which only use text. |
| Outcome: | The proposed models achieve state-of-the-art results on multilingual induction datasets even without help from linguistic knowledge or pretrained image encoders. |
Learning Visually Grounded Sentence Representations (N18-1)
Copied to clipboard
| Challenge: | Unsupervised sentence representation models suffer from the grounding problem because of lack of association between symbols and external information. |
| Approach: | They train a sentence encoder to predict image features of a caption and use them as sentence representations. |
| Outcome: | The proposed model improves on word embeddings and word representations on standard benchmarks. |
Unified Speech-Text Pre-training for Speech Translation and Recognition (2022.acl-long)
Copied to clipboard
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, Juan Pino
| Challenge: | Existing methods to pre-train speech and text use unlabeled data to learn universal feature representations. |
| Approach: | They propose a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. |
| Outcome: | The proposed method achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the Librispeech speech recognition task. |
Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing studies show that multitask learning improves speech translation performance by utilizing word embedding as the intermediate. |
| Approach: | They propose to use word embedding as an intermediate to improve multitask ST models by utilizing word embeds as input. |
| Outcome: | The proposed model outperforms existing models with sufficient training data but is still lacking in the low-resource scenario. |
Injecting structural hints: Using language models to study inductive biases in language learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study examines the cognitive inductive biases that make language learning possible. |
| Approach: | They structurally bias transformer language models by pretraining on synthetic data . they then evaluate their inductive biases by fine-tuning on three different languages . |
| Outcome: | The proposed method predisposes transformer models to three types of inductive biases . it also fine-tunes the models on three typologically-distant human languages . |
Grounding language acquisition by training semantic parsers using captioned videos (D18-1)
Copied to clipboard
| Challenge: | a new method for parsing sentences using captioned videos is being developed . we use video clips to ground the semantics of language, but without annotations . |
| Approach: | They develop a semantic parser that is trained in a grounded setting using captioned videos . they use a corpus of sentences paired with videos without other annotations to train it . |
| Outcome: | The proposed parser recovers the meaning of English sentences despite no annotations . learning a grounded semantic parsers can expand the range of data that parseurs can be trained on . |
On Generative Spoken Language Modeling from Raw Audio (2021.tacl-1)
Copied to clipboard
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, Emmanuel Dupoux
| Challenge: | Using a set of metrics to evaluate the learned representations, we aim to create a system that learns from natural interactions as infants learn their first language. |
| Approach: | They propose a task of learning acoustic and linguistic characteristics from raw audio and a set of metrics to evaluate the learned representations at acustic, linguistic and encoding levels. |
| Outcome: | The proposed models evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. |
How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases (2023.acl-long)
Copied to clipboard
| Challenge: | a recent study found that pre-training can teach language models to rely on hierarchical syntactic features . aaron ramirez: we find that pretraining on simpler language induces a hierarchic bias . |
| Approach: | They find that pre-training can teach language models to rely on hierarchical syntactic features . authors: this suggests that in cognitively plausible language acquisition settings, models may be more data-efficient . |
| Outcome: | a recent study shows that pre-training can teach language models to rely on hierarchical features . the findings suggest that in plausible language acquisition settings, language models may be more data-efficient than previously thought . |
A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning (2022.aacl-main)
Copied to clipboard
| Challenge: | Large-scale, pretrained vision-language models are growing in popularity due to impressive performance on downstream tasks with minimal finetuning. |
| Approach: | They propose to apply ranking metrics to image-text representations to investigate bias measures and debiasing methods to reduce various bias measures. |
| Outcome: | The proposed model reduces bias measures with minimal degradation to image-text representations. |