Papers by Florian Metze
Zero-shot Learning for Grapheme to Phoneme Conversion with Language Ensemble (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing work focuses on low-resource and endangered languages with limited training sets. |
| Approach: | They propose a hypothesis set for any unseen target language and combine it with a confusion network to propose 'the most likely hypothesis' they test the approach on over 600 unseened languages and demonstrate it significantly outperforms baselines. |
| Outcome: | The proposed model outperforms baselines on over 600 unseen languages. |
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | a new study examines zero-shot cross-lingual transfer of vision-language models . we study multilingual text-to-video search in non-English languages without annotations . |
| Approach: | They propose a Transformer-based model that learns contextual multilingual multimodal embeddings . they propose 'zero-shot cross-lingual transfer' to improve multilingual search . |
| Outcome: | The proposed model outperforms baselines on multilingual text-to-video search and multilingual image search on VTT and VATEX. |
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Combining visual modality with pretrained language models has been effective for descriptive tasks such as image captioning. |
| Approach: | They ask: do multimodal models combine visual and visual adapted language models? they find that CLIP image representations and scaling of language models do not consistently improve self-rationalization in multimodal tasks. |
| Outcome: | The proposed model types do not consistently improve self-rationalization in multimodal tasks. |
Acoustic-to-Word Models with Conversational Context Information (N19-1)
Copied to clipboard
| Challenge: | Existing speech recognition models are built at a sentence level, and therefore it may not capture conversational context information. |
| Approach: | They propose a direct acoustic-to-word, end-to end speech recognition model that integrates a conversational context with other available information and directly recognizes words from speech. |
| Outcome: | The proposed model outperforms a standard end-to-end speech recognition system on the Switchboard conversational speech corpus and shows that it is more accurate than existing models. |
Multimodal Abstractive Summarization for How2 Videos (P19-1)
Copied to clipboard
| Challenge: | Abstractive summarization is a task of producing a shorter version of the content in the document while preserving its information. |
| Approach: | They propose a new evaluation metric that measures semantic adequacy rather than fluency of abstractive summarization tasks. |
| Outcome: | The proposed model integrates information from different sources into a coherent output. |
Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion (P19-1)
Copied to clipboard
| Challenge: | Existing speech recognition systems are built at individual, isolated utterance level to make building systems computationally feasible. |
| Approach: | They propose to use text-based external word and/or sentence embeddings to integrate conversational context information into a single neural network model. |
| Outcome: | The proposed model outperforms standard end-to-end speech recognition models on the Switchboard conversational speech corpus and improves word error rate with better conversational-context representation. |
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021.findings-acl)
Copied to clipboard
Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke Zettlemoyer
| Challenge: | Existing methods for multimodal video understanding are task-specific, limiting their use for retrieval-style end tasks. |
| Approach: | They propose a task-agnostic multimodal pre-training approach that can accept video or text input, or both, for a variety of end tasks. |
| Outcome: | The proposed approach outperforms existing methods on a wider range of tasks while maintaining separability. |
Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning (2026.eacl-industry)
Copied to clipboard
Minseok Kim, Jingxiang Chen, Seong-Gyun Leem, Yin Huang, Rashi Rungta, Zhicheng Ouyang, Haibin Wu, Surya Teja Appini, Ankur Bansal, Yang Bai, Yue Liu, Florian Metze, Ahmed A Aly, Anuj Kumar, Ariya Rastrow, Zhaojiang Lin
| Challenge: | Using paralinguistic cues is challenging for speech large language models, authors say . limited training data, annotation difficulty, and models exploiting lexical shortcuts are challenges . a recent study shows that modeling paralinguistic reasoning with multitask RL improves paralinguistics understanding . |
| Approach: | They propose multi-task reinforcement learning with chain-of-thought prompting that elicits explicit affective reasoning. |
| Outcome: | The proposed model improves paralinguistics understanding over baselines and strong proprietary models by 8-12% on Expresso, IEMOCAP, and RAVDESS. |
Annotating High-Level Structures of Short Stories and Personal Anecdotes (L18-1)
Copied to clipboard
| Challenge: | Existing theories for narrative structures have been challenging to operationalize . authors present an annotation scheme to help computer systems understand stories better . |
| Approach: | They propose to consolidate and extend existing narratological theories and an annotation scheme . they will support an approach that enables systems to intelligently sustain complex communications with humans . |
| Outcome: | The proposed method consolidates and extends existing narratological theories . it will support an approach that enables systems to intelligently sustain complex communications with humans . |
Fine-Grained Grounding for Multimodal Speech Recognition (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models rely on global visual features that represent the entire image, but localizing the relevant regions of the image will make it possible to recover a larger set of words, such as adjectives and verbs. |
| Approach: | They propose a multimodal automatic speech recognition system that uses visual information from different parts of the image to ground the speech in the visual context. |
| Outcome: | The proposed model improves over approaches that use global visual features and localizes the correct proposals. |
CTC Alignments Improve Autoregressive Translation (2023.eacl-main)
Copied to clipboard
Brian Yan, Siddharth Dalmia, Yosuke Higuchi, Graham Neubig, Florian Metze, Alan W Black, Shinji Watanabe
| Challenge: | Connectionist Temporal Classification (CTC) is widely used for automatic speech recognition (ASR) but lags behind attentional decoder approaches in terms of translation quality. |
| Approach: | They propose to use a CTC/attention framework to validate this hypothesis by modifying the Hybrid CTC-Attention model proposed for automatic speech recognition to support text-to-text translation (MT) and speech-totext translation. |
| Outcome: | The proposed model outperforms pure-attention baselines across six translation tasks. |
Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks (2021.naacl-main)
Copied to clipboard
| Challenge: | ESPnet framework exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks. |
| Approach: | They propose a framework that exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks. |
| Outcome: | The proposed framework outperforms the state-of-the-art on speech translation tasks by +6 and +3 BLEU on the two test sets of Fisher-CallHome and +4 BLUE on the English-German and English-French test sets. |
Phone Inventories and Recognition for Every Language (2022.lrec-1)
Copied to clipboard
| Challenge: | Identifying phone inventories is crucial component in language documentation and preservation of endangered languages. |
| Approach: | They propose a probabilistic and non-probabilistic phone inventory model that estimates the phone inventory for any language listed in Glottolog. |
| Outcome: | The proposed model outperforms baseline models by 6.5 F1 and improves the PER (phone error rate) in phone recognition by 25%. |
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021.emnlp-main)
Copied to clipboard
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, Christoph Feichtenhofer
| Challenge: | Recent work adopts a "pre-training + fine-tuning" approach for zero-shot transfer to end tasks without fine- tuning. |
| Approach: | They propose a contrastive approach to pre-train a transformer model for zero-shot video and text understanding without using any labels on downstream tasks. |
| Outcome: | The proposed model outperforms supervised approaches on downstream tasks and outperformed previous approaches. |
Normalized Contrastive Learning for Text-Video Retrieval (2022.emnlp-main)
Copied to clipboard
| Challenge: | Cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance. |
| Approach: | They propose a normalized contrastive learning algorithm that normalizes the sum retrieval probabilities of each instance so that every text and video instance is fairly represented. |
| Outcome: | Empirical results show that NCL brings significant gains in text-video retrieval on different model architectures without any architecture engineering. |
NoiseQA: Challenge Set Evaluation for User-Centric Question Answering (2021.eacl-main)
Copied to clipboard
| Challenge: | Question-Answering (QA) systems are deployed in the real world . a lack of research attention has been devoted to studying the issues that arise when people use QA systems. |
| Approach: | They show that component components that precede an answering engine can introduce varied and considerable sources of error. |
| Outcome: | The proposed evaluations highlight the need for QA evaluation to expand to consider real-world use. |
AlloVera: A Multilingual Allophone Database (2020.lrec-1)
Copied to clipboard
David R. Mortensen, Xinjian Li, Patrick Littell, Alexis Michaud, Shruti Rijhwani, Antonios Anastasopoulos, Alan W Black, Florian Metze, Graham Neubig
| Challenge: | Phonemes are contrastive phonological units, and allophones are their various concrete realizations. |
| Approach: | They propose a resource that maps allophones to phonemes for 14 languages . they propose phonological representations that are much closer to a universal transcription . |
| Outcome: | The proposed resource maps from 218 allophones to phonemes for 14 languages. |
On Long-Tailed Phenomena in Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art Neural Machine Translation models struggle with generating low-frequency tokens, tackling which remains a major challenge. |
| Approach: | They propose a loss function to better adapt model training to structural dependencies of conditional text generation by incorporating inductive biases of beam search into the training process. |
| Outcome: | The proposed method leads to significant gains over cross-entropy across different language pairs, especially on the generation of low-frequency words. |
Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end spoken language understanding systems model sequence labeling as a sequence prediction task causing a divergence from its well-established token-level tagging formulation. |
| Approach: | They propose to model sequence labeling as a sequence prediction task . their systems explicitly separate the added complexity of recognizing spoken mentions from the NLU task of sequence labelling . |
| Outcome: | The proposed systems outperform both cascaded and direct models on a labeling task of named entity recognition across SLU benchmarks. |