Papers by Abdelrahman Mohamed
Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks (2024.acl-long)
Copied to clipboard
Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, Muhammad Abdul-Mageed
| Challenge: | MLLMs have proven effective in a wide range of tasks that require complex reasoning and linguistic comprehension, but they are limited to English-based settings. |
| Approach: | They propose a family of Arabic multimodal large language models with strong vision and language capabilities. |
| Outcome: | The proposed models show strong performance on visual reasoning tasks and language capabilities. |
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities (2022.acl-long)
Copied to clipboard
Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang, Zili Huang, Kushal Lakhotia, Shu-wen Yang, Shuyan Dong, Andy Liu, Cheng-I Lai, Jiatong Shi, Xuankai Chang, Phil Hall, Hsuan-Jui Chen, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, Hung-yi Lee
| Challenge: | Existing evaluation methods for transfer learning are limited in speech research . authors show that pre-trained models transfer well across multiple tasks . |
| Approach: | They propose a benchmark to evaluate pre-trained models by increasing task diversity and difficulty over SUPERB. |
| Outcome: | The proposed benchmark increases task diversity and difficulty over SUPERB-SG. |
Instruction-Guided Poetry Generation in Arabic and Its Dialects (2026.findings-acl)
Copied to clipboard
Abdelrahman Sadallah, Kareem Elozeiri, Mervat Abassy, Rania Elbadry, Mohamed Anwar, Abed Alhakim Freihat, Preslav Nakov, Fajri Koto
| Challenge: | Existing literature on Arabic poetry has focused on analysis tasks such as interpretation or metadata prediction, e.g., rhyme schemes and titles. |
| Approach: | They propose to use a large-scale instruction-based dataset to generate Arabic poetry based on predefined criteria such as style and rhyme . |
| Outcome: | The proposed model can generate poetry that is aligned with user requirements, based on automated metrics and human evaluation with native Arabic speakers. |
VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild (2024.acl-long)
Copied to clipboard
| Challenge: | VoiceCraft is a token-infilling neural codec language model for speech editing and zero-shot text-to-speech evaluation. |
| Approach: | They introduce a token infilling neural codec language model that performs on speech editing and zero-shot text-to-speech tasks. |
| Outcome: | The proposed model outperforms previous models on speech editing and zero-shot text-to-speech tasks. |
DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon (2022.tacl-1)
Copied to clipboard
Robin Algayres, Tristan Ricoul, Julien Karadayi, Hugo Laurençon, Salah Zaiem, Abdelrahman Mohamed, Benoît Sagot, Emmanuel Dupoux
| Challenge: | Existing nonparametric models for text segmentation use a Dirichlet process to jointly segment sentences and build a lexicon of word types. |
| Approach: | They propose a Bayesian nonparametric model that uses a Dirichlet process to jointly segment sentences and build a lexicon of word types. |
| Outcome: | The proposed model improves on the Zero Resource Speech Benchmark 2017 and can learn semantic and syntactic representations as assessed by a new spoken word embedding benchmark. |
Text-Free Prosody-Aware Generative Spoken Language Modeling (2022.acl-long)
Copied to clipboard
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, Wei-Ning Hsu
| Challenge: | Experimental results show that generative spoken language models (LMs) are natural unsupervised multitask learners. |
| Approach: | They propose a prosody-aware generative spoken language model that uses discovered units to generate natural, meaningful, and coherent speech. |
| Outcome: | The proposed model can generate natural, meaningful, and coherent speech given a spoken prompt. |
JEEM: Vision-Language Understanding in Four Arabic Dialects (2026.findings-eacl)
Copied to clipboard
Karima Kadaoui, Hanin Atwany, Hamdan Al-Ali, Abdelrahman Mohamed, Ali Mekky, Sergei Tilga, Natalia Fedorova, Ekaterina Artemova, Hanan Aldarmaki, Yova Kementchedjhieva
| Challenge: | Existing evaluation datasets feature Western-centric images and English text, while their non-English counterparts are often derived from the latter. |
| Approach: | They propose to evaluate Vision-Language Models (VLMs) on visual understanding across four Arabic-speaking countries: Jordan, The Emirates, Egypt, and Morocco. |
| Outcome: | The proposed model underperforms in visual understanding and dialect-specific generation across four Arabic-speaking countries. |
textless-lib: a Library for Textless Spoken Language Processing (2022.naacl-demo)
Copied to clipboard
Eugene Kharitonov, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Paden Tomasello, Ann Lee, Ali Elkahky, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, Yossi Adi
| Challenge: | Textless spoken language processing is an exciting area of research that promises to extend applicability of the standard NLP toolset onto spoken language and languages with few or no textual resources. |
| Approach: | They introduce textless-lib, a PyTorch-based library that provides textless spoken language processing tools. |
| Outcome: | The proposed library significantly simplifies research in the textless setting and will be a handful for speech researchers and the NLP community at large. |
Generative Spoken Dialogue Language Modeling (2023.tacl-1)
Copied to clipboard
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdelrahman Mohamed, Emmanuel Dupoux
| Challenge: | dGSLM is the first “textless” model able to generate audio samples of naturalistic spoken dialogues. |
| Approach: | They propose a model that generates speech, laughter, and other paralinguistic signals in two channels simultaneously and reproduces more naturalistic turn taking compared to a text-based cascaded model. |
| Outcome: | The proposed model reproduces more naturalistic and fluid turn taking than a text-based cascaded model. |
Self-supervised Representation Learning for Speech Processing (2022.naacl-tutorials)
Copied to clipboard
Hung-yi Lee, Abdelrahman Mohamed, Shinji Watanabe, Tara Sainath, Karen Livescu, Shang-Wen Li, Shu-wen Yang, Katrin Kirchhoff
| Challenge: | Self-supervised representation learning (SSL) uses proxy supervised learning tasks to obtain training data from unlabeled corpora. |
| Approach: | They propose to survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing to scale up current machine learning technologies. |
| Outcome: | The proposed tutorial is highly relevant to the special theme of ACL about language diversity. |
Arabic Diacritization Using Morphologically Informed Character-Level Model (2024.lrec-main)
Copied to clipboard
Muhammad Morsy Elmallah, Mahmoud Reda, Kareem Darwish, Abdelrahman El-Sheikh, Ashraf Hatim Elneima, Murtadha Aljubran, Nouf Alsaeed, Reem Mohammed, Mohamed Al-Badrashiny
| Challenge: | Diacritics are typically omitted in Arabic writings and the reader needs to guess the proper diacritics as they are reading. |
| Approach: | They propose a morphologically informed character-level model that can recover both types of diacritics simultaneously. |
| Outcome: | The proposed model achieves lowest word-level diacritization error rate for Classical Arabic, MSA, and two dialectal Arabic texts. |
VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation (2026.findings-acl)
Copied to clipboard
| Challenge: | Neural codec language models (NCLMs) lack fine-grained controllability and inability to extrapolate to sequence lengths much longer than those seen during training. |
| Approach: | They propose a novel autoregressive encoder-decoder neural codec language model that can be trained with a Continuation-Prompt Mixed training system. |
| Outcome: | The proposed model outperforms or is on par with current state-of-the-art models on short-form benchmarks such as LibriSpeech and Seed-TTS in terms of intelligibility and naturalness. |
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020.acl-main)
Copied to clipboard
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer
| Challenge: | Recent work has shown gains by improving the distribution of masked tokens and the order in which mucked tokens are predicted. |
| Approach: | They propose a denoising autoencoder for pretraining sequence-to-sequence models that corrupts text with an arbitrary noising function and learns a model to reconstruct the original text. |
| Outcome: | The proposed model outperforms RoBERTa on GLUE and SQUAD and provides a 1.1 BLEU increase over a back-translation system for machine translation. |
Casablanca: Data and Models for Multidialectal Arabic Speech Recognition (2024.emnlp-main)
Copied to clipboard
Bashar Talafha, Karima Kadaoui, Samar Magdy, Mariem Habiboullah, Chafei Chafei, Ahmed El-Shangiti, Hiba Zayed, Mohamedou Tourad, Rahaf Alhamouri, Rwaa Assi, Aisha Alraeesi, Hour Mohamed, Fakhraddin Alwajih, Abdelrahman Mohamed, Abdellah El Mekki, El Moatez Billah Nagoudi, Benelhadj Saadia, Hamzah Alsayadi, Walid Al-Dhabyani, Sara Shatnawi, Yasir Ech-chammakhy, Amal Makouar, Yousra Berrachedi, Mustafa Jarrar, Shady Shehata, Ismail Berrada, Muhammad Abdul-Mageed
| Challenge: | despite recent advances in speech processing, the majority of world languages and dialects remain uncovered. |
| Approach: | They propose to collect and transcribe a new Arabic dataset for eight dialects . they also develop strong baselines exploiting the new dataset . |
| Outcome: | The proposed dataset covers eight Arabic dialects, including Algerian, Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, and Yemeni. |
LLMs Can Compensate for Deficiencies in Visual Representations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a strong language backbone in vision-language models compensates for weak visual features by contextualizing or enriching them. |
| Approach: | They investigate whether strong language backbone compensates for weak visual features . they use CLIP-based vision encoders to perform controlled self-attention ablations . |
| Outcome: | The proposed model compensates for weak visual features by contextualizing or enriching them. |
Textless Speech Emotion Conversion using Discrete & Decomposed Representations (2022.emnlp-main)
Copied to clipboard
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu Anh Nguyen, Morgan Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, Yossi Adi
| Challenge: | Existing methods for modifying emotion of speech are difficult because emotion affects all levels simultaneously. |
| Approach: | They propose a method to convert a spoken language speech into a model of emotion . they use phonetic-content units, prosodic features, speaker, and emotion to modify the emotion a speech utterance has. |
| Outcome: | The proposed method beats text-based systems in terms of perceived emotion and audio quality. |
On Generative Spoken Language Modeling from Raw Audio (2021.tacl-1)
Copied to clipboard
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, Emmanuel Dupoux
| Challenge: | Using a set of metrics to evaluate the learned representations, we aim to create a system that learns from natural interactions as infants learn their first language. |
| Approach: | They propose a task of learning acoustic and linguistic characteristics from raw audio and a set of metrics to evaluate the learned representations at acustic, linguistic and encoding levels. |
| Outcome: | The proposed models evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. |
Unified Speech-Text Pre-training for Speech Translation and Recognition (2022.acl-long)
Copied to clipboard
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, Juan Pino
| Challenge: | Existing methods to pre-train speech and text use unlabeled data to learn universal feature representations. |
| Approach: | They propose a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. |
| Outcome: | The proposed method achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the Librispeech speech recognition task. |